Educational Assessment, Evaluation, and Research Commons^™

Guidelines For Automatic Grading Of Student Essays Using Large Language Models, Diwakar Yalpi, Sruta Keerti Kasula, Ravi Mukkamala

Computer Science Faculty Publications

Automated essay evaluation using large language models (LLMs) has emerged as a promising approach to support scalable and consistent educational assessment. However, the effectiveness of LLM-based grading varies significantly across evaluation dimensions and is highly influenced by prompt design and model selection. In this study, we evaluate five state-of-the-art LLMs across five rubric-based categories: Relevance to Question, Reasoning and Critical Thinking, Evidence and Examples, Organization, and Clarity and Writing Quality. We systematically investigate the impact of three prompting strategies, including rubric-only prompting, exemplar-based prompting (with and without rubric guidance)(Original and Refined prompt designs) incorporating structured instructions. Additionally, a prompt ablation …

The (Non) Numeric Truths Behind College And Career Aspirations: A Quantcrit Analysis Of The Precarious Pathways For Secondary Students Of Color, Norma A. Marrun, Danielle Mireles, Jori S. Beck, Christine Clark

Teaching & Learning Faculty Publications

This article explored the college and career choices of 963 students across six urban secondary schools in a large public school district in the Southwestern United States. A Quantitative Critical Race Theory, or QuantCrit, analysis was applied to expose the false narrative of number neutrality and the systemic racism that belies it. Findings demonstrate how the myth of meritocracy through education (as ‘the great equalizer’) is perpetuated when educators hold onto racist and ableist beliefs about who can pursue college, and ignore the structural disadvantages underlying the college and career aspirations of students of Color juxtaposed with the reality of …

Redefining Educational Technology: A Critical Collaborative Inquiry, Aras Bozkurt, Helen Crompton, Robert Farrow, Agnes Kukulska-Hulme, Jon Dron, Richard West, Agnieszka (Aga) Palalas, Matth Bower, Junhong Xiao, Ahmed Tlili, Danah Henriksen, Angelica Pazurek, Henk Huijser, Thomas K. F. Chiu, Petar Jandrić, Katy Jordan, John Curry, Royce Kimmons, Mutlu Cukurova, Thomas Reeves, Gwo-Jen Hwang, Peter Shea, Jason Lodge, Martin Weller, Davy Ng, Tutaleni Iita Asino

STEMPS Faculty Publications

Educational technologists have not settled on a fixed definition of the field and likely never will. However, attempting to define the field helps to understand the epistemological meanings that shape what the field sees, values, and considers worth pursuing. Through a critical historical review spanning over a century, alongside theoretical engagement with the concepts of entanglement and distributed agency, this paper identifies three key insufficiencies in current educational technology frameworks. These are the persistence of an instrumental-facilitative paradigm that treats technology as a resource deployed by human agents; the theoretical dissolution of the pedagogy-technology dichotomy that existing definitions have not …

Considerations For School Leadership In Leveraging School-Based Behavior Analysis For Systemic Improvement In Public Education, Virginia Public Schools Behavior Analyst Network, Kira Austin, Elizabeth Boynton, Amanda K. Cash, Meredith Eads, Daniel Irwin, Selena Layden, Daria K. Lorio-Barsten, Emily Muise, Nadine Pittman, Felicia Shelton, Indya Watts-Graves

Human Movement Studies & Special Education Faculty Publications

Public schools face significant challenges, including rising student behavioral needs, chronic absenteeism, and high rates of teacher burnout. These issues are interconnected with and inextricably linked to a need for effective, evidence-based behavioral supports. Educators frequently enter the field lacking the necessary skills and training to manage complex student behavior, often resulting in a reliance on punitive practices that exacerbate problems.

This paper proposes that the expertise of school-based behavior analysts (SBBAs), who are Board Certified Behavior Analysts® (BCBAs®) specializing in education, offers a powerful, underutilized solution. SBBAs are trained in applied behavior analysis (ABA), a data-driven science focused on …

Exploring Student Confidence And Calibration Accuracy In College Algebra: A Mixed-Methods Explanatory Sequential Study, Courtney T. Hill

STEMPS Theses & Dissertations

Calibration, or students’ ability to predict and postdict their performance on an academic task, has important implications for academic performance. It can influence students' decision making processes concerning learning activities, as well as their levels of motivation and self-regulation (Kolovelonis, 2023). The literature on calibration is clear, often indicating that students are not great at these judgements (Lin et. al, 2001). While this phenomenon is well documented, it is not an easy one to explain. Therefore, a two-phase explanatory-sequential mixed method design was used to examine undergraduate students’ calibration judgements on a college algebra exam. To further contribute to the …

Analysis Of Industry Recognized Credentials That Lead To Quality Employment For High School Graduates, Teri L. Harris

Educational Leadership & Workforce Development Theses & Dissertations

The purpose of this study was to identify to what extent industry recognized credentials (IRCs) earned in secondary school lead to high quality employment for students after high school and which specific IRCs should be integrated into career and technical education (CTE) courses. This author sought to explore if CTE courses in high school could be the vehicle for students to earn IRCs that led to higher quality jobs, higher pay, and a job to support a family. The researcher gathered empirical evidence using existing data from a survey of business leaders in each of the nationally recognized 14 Career …

Am I As Effective At Identifying Emotions As Artificial Intelligence? A Comparative Study Of Emotional Recognition, Traci R. Grove, Alexandra T. Lucas, Maryann Martin, Cathleen M. Deckers, Lulu Sherif Mahmood, Nicole Danaher-Garcia, Mark W. Scerbo, Suzan Kardong-Edgren, Janice C. Palaganas

Psychology Faculty Publications

Background

Learning conversations, or dialogues aimed at deepening understanding and reflection, are deeply influenced by emotions. Effective communication is influenced by emotional intelligence - the ability to recognize, understand, and manage both one’s own and others’ emotions. While advances in artificial intelligence (AI) offer new tools for emotion recognition, these technologies still struggle with accurately interpreting subtle and culturally diverse emotional expressions, sparking debate about their reliability and effectiveness. This article provides a comparative analysis of human versus AI recognition of emotions during an end-of-course reflective learning conversation.

Methods

Emotions during a structured post-conference debriefing were analyzed and coded by …

A Comparison Of Service Needs For Face-To-Face To Online Programs, Ali Ardalan, Roya K. Ardalan, G. Steven Rhiel

Information Technology & Decision Sciences Faculty Publications

Higher education institutions strive to gain a competitive advantage over similar programs offered by other institutions. A criterion for comparing programs is the quality of service. The definition of quality from the customer's view is "fitness for use," which incorporates the value generated by the product. Each learner may evaluate programs differently from other learners. It is impossible to develop a unique standard for program quality that is defined by a specific set of factors and specific level for each of these factors. This study presents an extensive set of factors that may affect learners' satisfaction. Higher education administrators should …

Application Of Fuzzy Logic For Evaluation Of Student Performance, M. Tomovic, C. Tomovic, V. Jovanovic, S. Bawab

Engineering Technology Faculty Publications

The evaluation of student performance is a multifaceted process that traditionally relies on rigid grading systems, often failing to capture the complexity of student abilities and learning dynamics. This paper explores the application of fuzzy logic to enhance the accuracy and adaptability of student performance evaluation. By leveraging fuzzy logic principles, such as membership functions and linguistic variables, we model and assess the various factors influencing student performance—ranging from academic grades to engagement, participation, and personal learning styles. The approach allows for the representation of imprecise, uncertain, and subjective data, offering a more holistic view of a student’s progress. We …

The Impact Of Guided Notes Intervention On Learning Achievement, Quality Of Notes, And Perception, Bryan Richard Biggers

STEMPS Theses & Dissertations

Guided notes in the form of teacher-prepared materials that guide students through the lesson with cues and space have been recommended as a possible aid to support student learning. The present study investigated the results of guided notes on the students’ learning achievement, the quality of the students’ notes, and the students’ perceptions of note-taking. Sixty students were enrolled in an eight-week, asynchronous, online, general education course and were divided into nonintervention and intervention groups where the nonintervention groups took notes without support while the students in the intervention group took notes with guided notes during weeks two through seven. …

Multifaceted Challenges And Opportunities: Concurrent Mixed Methods Research To Investigate Chinese Exchange Students' Experiences In The Us Transnational Online Learning Ecology, Xinyue Ren, Yi Zhou

STEMPS Faculty Publications

Transnational online education has become an emerging trend in promoting academic collaboration and exchange in higher education during and after the pandemic. Guided by the learning ecology framework, we conducted concurrent mixed methods research by distributing an online survey to Chinese students (N=51) to collect quantitative and qualitative data. This helped us deeply understand their experiences and perceptions of a cross-cultural online learning environment in the United States. We applied multiple regression and thematic analysis to analyze the data. The merging of both types of data analysis indicated the confirmation of the importance of discipline-based English learning environments, the transformation …

Caregivers' Perceptions Of Covid-19 Educational Disruptions On Children With Developmental Language Disorder And Typically Developing Peers, Katherine Radville, Danika Pfeiffer, Karynn Sheranian, Julie Wolter, Jessie Ricketts, Tiffany L. Hogan

Communication Disorders & Special Education Faculty Publications

Purpose: Understanding the experiences of families of children with developmental language disorder (DLD) during COVID-19 educational disruptions is essential for designing responsive supports during pandemic recovery efforts and beyond. This qualitative study describes the experiences of families of first-and second-grade children with DLD during the pandemic as compared to the experiences of families of typically developing (TD) peers.

Method: A conventional content analysis approach was used to analyze caregivers’ written responses to open-ended questions regarding their perceptions of COVID-19 educational disruptions. Responses were analyzed separately by group: caregivers of children with DLD (n= 23) and caregivers of TD children (n= …

The Underrepresentation Of Black Females In Cybersecurity, Makendra Latrice Crosby

Cybersecurity Undergraduate Research Showcase

The significance of cybersecurity methods, strategies, and programs in protecting computers and electronic devices is crucial throughout the technological infrastructure. Despite the considerable growth in the cybersecurity field and its expansive workforce, there exists a notable underrepresentation, specifically among Black/African American females. This study examines the barriers hindering the inclusion of Black women in the cybersecurity workforce such as socioeconomic factors, limited educational access, biases, and workplace culture. The urgency of addressing these challenges calls for solutions such as education programs, mentorship initiatives, creating inclusive workplace environments, and promoting advocacy and increased awareness within the cybersecurity field. Additionally, this paper …

Report On Odu Findings From The Virginia Course Materials Survey Fall 2021, Karen Vaughan

Libraries Reports and Documents

From Introduction:

From October 1-December 31, 2021, the Virtual Library of Virginia (VIVA) conducted a survey of students at Virginia colleges and universities. More than 5,600 valid student responses from 41 institutions were received, reflecting an overall response rate of 10%. The largest response (58.5%) was from doctoral institutions, with private, two-year, and four-year institutions responding at 12-14%. A random sample of ODU students produced 287 responses (see Appendix A for ODU demographics). The survey aimed to answer these questions:

• What is the impact of course material costs on educational equity among Virginia students?
• What course content materials do students …

Unveiling Pre-Service Teachers' Epistemological Beliefs In The Planning Of Korean Language Assessment, Haeju Cheon, Seongryeong Yu

Teaching & Learning Faculty Publications

This study explores the epistemological beliefs of pre-service elementary teachers in Korean language education and their impact on performance assessment task planning. Grounded in the correlation between teachers’ beliefs and instructional approaches, the research aims to redefine Schommer’s (1990) epistemological dimensions in tasks designed by pre-service teachers. Data from 150 participants were analyzed quantitatively and qualitatively, revealing three belief-based clusters: Naive (24.7%), Mixed (51.3%), and Sophisticated (24.0%). Task planning patterns in all groups focus on individual perspective analysis and selfreflection. The Mixed group emphasizes related readings, exposing participants to various perspectives. In contrast, the Naive group employs diverse tasks in …

Authentic Learning Design Failures: The Need For Learner And Contextual Analysis And Participatory Design, Mohan Yang, Jon Harbor

STEMPS Faculty Publications

This design case examines what program leaders learned from failures in the design of a program of authentic learning about teaching diverse audiences through educational outreach. The program was initiated and then redesigned to develop the teaching and communication skills of graduate students from a wide range of backgrounds by engaging them in authentic experiences with middle school teachers and students. Analysis of post-program data revealed seven design failures related to the lack of upfront analysis to inform the program design. Each design failure was detailed through a fishbone diagram method to identify the corre- sponding contributing factors. The failures …

Improving Reliability In Assessing Integrative Learning Using Rubrics: Does Group Norming Help?, Lanah Stafford, Erin Cousins, Linda Bol, Megan Mize

STEMPS Faculty Publications

Integrative learning is an important outcome for graduates of higher education. Therefore, it should be well-defined and assessed reliably. The American Association of Colleges & Universities has developed a rubric to define and assess integrative learning, but it has low reliability. This pilot study examines whether this rubric's reliability can be improved by training users on how to use the rubric in a group setting rather than individually. Twelve faculty were trained to score undergraduate ePortfolios using the Integrative and Applied Learning VALUE Rubric. Half of the faculty were trained in an individual setting and half in a group setting …

Conducting A Formative Evaluation On A Course-Level Learning Analytics Implementation Through The Lens Of Self-Regulated Learning And Higher-Order Thinking, Pauline S. Muljana, Tian Luo, Greg Placencia

STEMPS Faculty Publications

Self-regulated learning (SRL) and higher-order thinking skills (HOTS) are associated with academic achievement, but fostering these skills is not easy. Scholars have suggested an alternative way to scaffold these important skills through learning analytics (LA). This paper presents a formative evaluation of a course-level LA implementation through the lens of self-regulated learning (SRL) and higher-order thinking skills (HOTS). We explored the changes in students’ SRL, HOTS, and perceptions at the end of the course term. Results indicate an increase in some elements of SRL and HOTS, and positive student perceptions. Discussion on implications and opportunities for informing future teaching strategies …

Determining The Difficulty And Discrimination Parameters Of A Mathematics Performance-Based Assessment, Abraham Gyamfi, Douglas G. Wren

Educational Leadership & Workforce Development Faculty Publications

Performance-based assessment (PBA) is different from traditional testing methods in that PBA presents real-life problems for students to solve by integrating critical thinking with their content knowledge and skills. Implementing PBA regularly in mathematics classes is associated with improved student achievement and motivation to learn; however, there are concerns about the general lack of psychometric data to support the use of performance assessments. To address such concerns, this study applied item response theory to estimate the difficulty and discrimination indices of items that comprised a newly developed mathematics PBA. Data were collected by administering the PBA to 750 senior high …

Leading Through Partnership: An Examination Of Longitudinal Trends In A School-University Partnership, Kala Burrell-Craft, Patrick Doyle, Mary Gregory, Jori S. Beck, Peter D. Wiens

Teaching & Learning Faculty Publications

The purpose of this manuscript is to detail pre- and post- changes that occurred after a principal, Dr. D, was moved from one middle school to another. Dr. D brought an established partnership with him to his new school including a teacher residency program, UTeach replication program, and an on-site course. The results of ANOVAs are presented regarding the school climate according to student and teacher perspectives. Significant results were found on a number of variables including student belonging, bullying, and teacher autonomy. This manuscript addresses the National Association for Professional Development Schools Essential #5: Research and Results.

Making A Difference Through Sustained In-Service Teacher Training, Abha Gupta, Guang Lea Lee

Teaching & Learning Faculty Publications

This study is based on collaboration between a school and a university on professional development training of 4th and 5th grade elementary school teachers in a southeastern state in the USA. The study was three-pronged and focused on teacher knowledge, pedagogy, and student achievement. We examined how the building of teacher capacity affected the performance of underachieving students in math and literacy. Underachieving students were targeted with specific strategies, projects, problems solving stories, self-reflection, and higher-level thinking questions. Student performance was measured for literacy achievement, with quantitative and qualitative measures used for data collection purposes. Students showed progress over previous …

Reconfigurable Modules For Automatic Creating Robotic Courses, Ana Djuric, Vukica Jovanovic

Engineering Technology Faculty Publications

The main objective of this paper is to describe a teaching tool that can be used for automated creation of different problems that can be used for assessing student knowledge while teaching industrial robotics courses. These materials can be used for instructors who need to create customized questions and answers for different student homework. Wayne State University, Detroit, MI offer different industrial robotics and courses related to fundamental robotic theory: kinematics, dynamics, and control. Old Dominion University, Norfolk, VA offers the course in Introduction to Industrial Robotics. Both programs are Mechanical Engineering Technology under the Engineering Technology departments / division. …

Creating Conditions For Collaboration: A Case Study On Design-Based Professional Learning, Leslie A. Lehner

Educational Leadership & Workforce Development Theses & Dissertations

Design thinking is a process for solving complex problems through creative actions. There are three primary ways design thinking is being used in education today: (a) as a teaching tool, (b) as a strategy for teachers to redesign their students’ learning experiences, and (c) as a tool for school leaders to design school change (Gallagher & Thordarson, 2018). Design thinking includes a strong emphasis on team-based learning regarding the problem and possible solutions (Lindberg et al., 2010). This case study describes one school’s approach to navigating complex challenges, inspiring school change, and developing conditions for meaningful collaboration using a design …

The Influence Of Participation In A Multi-Disciplinary Collaborative Service Learning Project On The Effectiveness Of Team Members In A 100-Level Mechanical Engineering Class, Stacie I. Ringleb, Pilar Pazos, Julia Noginova, Francisco Cima, Orlando Ayala, Krishnanand Kaipa, Jennifer Kidd, Kristie Gutierrez

Mechanical & Aerospace Engineering Faculty Publications

Engineers need to develop professional skills, including the ability to work successfully in teams and to communicate within and outside of their discipline, in addition to required technical skills. A collaborative multi-disciplinary service learning project referred to as Ed+gineering was implemented in a 100-level mechanical engineering course. In this collaboration, mechanical engineering students, primarily in the second semester of their freshman year or first semester of their second year, worked over the course of a semester with education students taking a foundations course to develop and deliver engineering lessons to fourth or fifth graders. Students in comparison engineering classes worked …

Review Of: Estrellado, R. A., Freer, E. A., Motsipak, J., Rosenberg, J. M., & Velásquez, I. C. (2021). Data Science In Education Using R. London, England: Routledge, Esther Michela, Robert L. Moore

STEMPS Faculty Publications

Data use in education is continually expanding in scope (Romero and Ventura 2020), and though programming resources are plentiful, those that truly scaffold the learning process are scarce. Data Science in Education Using R by Estrellado, Freer, Motsipak, Rosenberg, and Velásquez is a combination primer and workbook. This format allows the volume to serve as both an introductory R resource aimed to help readers overcome the daunting initial roadblocks to proficiency and familiarize them with educational data uses. We reviewed this book from the perspective of a beginning R user with little other programming background for whom this book is …

Investigating Pre-Service Teachers’ Perceptions Of The Virginia Computer Science Standards Of Learning: A Qualitative Multiple Case Study, Valerie Sledd Taylor

Educational Leadership & Workforce Development Theses & Dissertations

Computer science education is being recognized globally as necessary to better prepare students in all grade levels, K-12, for future success. As a result of this focus on computer science education in the United States and around the world, there is an increased demand for highly qualified teachers with content and pedagogical knowledge to successfully support student learning. As a result, there is a call to include and improve the computer science training offered to pre-service teachers in their educator preparation programs from methods courses to practicum and student teaching experiences. Thus, it is important to understand how pre-service teachers …

Standardizing America: Why It Should Be A Method Of The Past, Samantha N. Jackson

OUR Journal: ODU Undergraduate Research Journal

This paper examines, critiques, and suggests improvements on the method of standardized testing in American schools. This paper discusses the history and development of standardized testing and its initial purpose and intentions. Additionally, the effects of standardized testing on students, teachers, and parents are evaluated, with special consideration on how high stakes testing adversely affects disadvantaged student groups such as children in minorities and low-income districts, bilingual students, and children with disabilities. The research suggests that standardized testing is not only damaging to students in these groups, but most likely not the most efficient way of testing student performance in …

Using Self-Based Methodologies To Unpack Mathematics Teacher Educators' Work, Elizabeth Suazo-Flores, Signe E. Kastberg, Melva R. Grant, Jennifer Ward, Sue Ellen Richardson, Olive Chapman

Teaching & Learning Faculty Publications

Narrative inquiry, self-study, and autoethnography (i.e., self-based methodologies) are methodologies used by mathematics teacher educators (MTEs). These methodologies have opened up the field by unpacking and unearthing MTEs' work communicating findings from their practices. Building from our previous working groups at PME-NA 2018-2020, we sustain a community where MTEs can feel supported in their study design, implementation, representation of findings, and publication using self-based methodologies. At PME-NA Philadelphia, we will continue our work at PME-NA Mexico on self-based methodologies to develop perspectives on philosophical underpinnings of self-based methodologies and addressing trustworthiness and authenticity in our reports.

A Continuum Of Data Literacy For Teaching, Jori S. Beck, Diana Nunnaley

Teaching & Learning Faculty Publications

Accountability for education in the United States has existed since the first formalized system of education. Although this accountability is an important part of society, these systems must be expanded beyond high stakes assessments to include other types of data including P-12 student voices. The purpose of the current manuscript is to present a continuum for data literacy for teachers that spans preservice to inservice teacher education. We conceptualize data literacy for teachers as a metaconstruct that includes the construct of assessment literacy. The research on enabling and marginalizing factors and exposure to data are reviewed at the preservice and …

A Continuum Of Critical Consciousness: Exploring One Resident's Concerns, Jori S. Beck

Teaching & Learning Faculty Publications

In the United States, there is a demand for richer clinical teacher education experiences. Partially in response to this call, innovative new programs like teacher residencies are being developed. As teacher preparation programs are shaped by these mandates, researchers must respond to shifts in the field. The current manuscript includes data from a resident, or teacher candidate, enrolled in a residency program—specifically, his yearlong apprenticeship. Using interviews and other qualitative data, the author examined how the resident’s concerns shifted and also how these concerns differed from those uncovered in previous research. Specifically, his concerns appeared to be more dynamic than …