Educational Assessment, Evaluation, and Research Commons^™

Mokken Scale Analysis Of The Teacher Formative Assessment Practice Scale, Eric Asare

Knowledge and Creativity Expo

This study examined the nonparametric item response characteristics of the Teacher Formative Assessment Practice Scale (TFAPS) using Mokken scale analysis within its established two‑factor structure. The TFAPS includes a teacher‑directed subscale (items 1–6) and a student‑directed subscale (items 7–10), each intended to capture distinct dimensions of formative assessment practice. Weighted Loevinger scalability coefficients for polytomous items were computed to assess how well each subscale met the assumptions of the monotone homogeneity model. Results indicated that the teacher‑directed subscale had a scalability coefficient of H = 0.354 and Cronbach’s alpha of 0.721, suggesting weak but acceptable internal structure. In contrast, the …

Guidelines For Automatic Grading Of Student Essays Using Large Language Models, Diwakar Yalpi, Sruta Keerti Kasula, Ravi Mukkamala

Computer Science Faculty Publications

Automated essay evaluation using large language models (LLMs) has emerged as a promising approach to support scalable and consistent educational assessment. However, the effectiveness of LLM-based grading varies significantly across evaluation dimensions and is highly influenced by prompt design and model selection. In this study, we evaluate five state-of-the-art LLMs across five rubric-based categories: Relevance to Question, Reasoning and Critical Thinking, Evidence and Examples, Organization, and Clarity and Writing Quality. We systematically investigate the impact of three prompting strategies, including rubric-only prompting, exemplar-based prompting (with and without rubric guidance)(Original and Refined prompt designs) incorporating structured instructions. Additionally, a prompt ablation …

The (Non) Numeric Truths Behind College And Career Aspirations: A Quantcrit Analysis Of The Precarious Pathways For Secondary Students Of Color, Norma A. Marrun, Danielle Mireles, Jori S. Beck, Christine Clark

Teaching & Learning Faculty Publications

This article explored the college and career choices of 963 students across six urban secondary schools in a large public school district in the Southwestern United States. A Quantitative Critical Race Theory, or QuantCrit, analysis was applied to expose the false narrative of number neutrality and the systemic racism that belies it. Findings demonstrate how the myth of meritocracy through education (as ‘the great equalizer’) is perpetuated when educators hold onto racist and ableist beliefs about who can pursue college, and ignore the structural disadvantages underlying the college and career aspirations of students of Color juxtaposed with the reality of …

Redefining Educational Technology: A Critical Collaborative Inquiry, Aras Bozkurt, Helen Crompton, Robert Farrow, Agnes Kukulska-Hulme, Jon Dron, Richard West, Agnieszka (Aga) Palalas, Matth Bower, Junhong Xiao, Ahmed Tlili, Danah Henriksen, Angelica Pazurek, Henk Huijser, Thomas K. F. Chiu, Petar Jandrić, Katy Jordan, John Curry, Royce Kimmons, Mutlu Cukurova, Thomas Reeves, Gwo-Jen Hwang, Peter Shea, Jason Lodge, Martin Weller, Davy Ng, Tutaleni Iita Asino

STEMPS Faculty Publications

Educational technologists have not settled on a fixed definition of the field and likely never will. However, attempting to define the field helps to understand the epistemological meanings that shape what the field sees, values, and considers worth pursuing. Through a critical historical review spanning over a century, alongside theoretical engagement with the concepts of entanglement and distributed agency, this paper identifies three key insufficiencies in current educational technology frameworks. These are the persistence of an instrumental-facilitative paradigm that treats technology as a resource deployed by human agents; the theoretical dissolution of the pedagogy-technology dichotomy that existing definitions have not …

How To Create Accessible Osce Exams For Learners With Disabilities, Lisa M. Meeks, Jordan Juliao, Emily Langham, Anne Lewis, Judith Brenner

University Administration Publications

[Introduction] Objective structured clinical examinations (OSCEs) are widely used across health professions education to assess learners' clinical skills through structured encounters with standardised patients [1]. OSCEs also present a known barrier to disability access for medical students with disabilities [2-6], a rapidly growing population. Using trained actors, also known as standardised patients who portray the same clinical scenario for all learners, these examinations allow for assessment of medical knowledge, clinical reasoning, communication skills and elements of the physical examination in a controlled and standardised setting [1]. They may be used for formative feedback, summative assessment or progression decisions, and they …

Ensuring Rich Rigor Of Qualitative Methodologies In Behavior Analytic Research, Daria K. Lorio-Barsten, Selena J. Layden

Human Movement Studies & Special Education Faculty Publications

Quantitative methods remain the hallmark of research in applied behavior analysis. Yet, such methods frequently fail to capture the nuances of context where behavior analysis is practiced. Therefore, qualitative methods can provide complementary means to gain deeper insight into changes in socially significant behavior. We believe that researchers within the field of behavior analysis have much to gain from embracing qualitative methodologies. We propose that more researchers can and should consider conducting rigorous qualitative research to elevate the voices of the participants and relate the depth and complexities of their nuanced experiences. This article discusses Tracy’s “big tent” quality criteria …

Considerations For School Leadership In Leveraging School-Based Behavior Analysis For Systemic Improvement In Public Education, Virginia Public Schools Behavior Analyst Network, Kira Austin, Elizabeth Boynton, Amanda K. Cash, Meredith Eads, Daniel Irwin, Selena Layden, Daria K. Lorio-Barsten, Emily Muise, Nadine Pittman, Felicia Shelton, Indya Watts-Graves

Human Movement Studies & Special Education Faculty Publications

Public schools face significant challenges, including rising student behavioral needs, chronic absenteeism, and high rates of teacher burnout. These issues are interconnected with and inextricably linked to a need for effective, evidence-based behavioral supports. Educators frequently enter the field lacking the necessary skills and training to manage complex student behavior, often resulting in a reliance on punitive practices that exacerbate problems.

This paper proposes that the expertise of school-based behavior analysts (SBBAs), who are Board Certified Behavior Analysts® (BCBAs®) specializing in education, offers a powerful, underutilized solution. SBBAs are trained in applied behavior analysis (ABA), a data-driven science focused on …

Exploring Student Confidence And Calibration Accuracy In College Algebra: A Mixed-Methods Explanatory Sequential Study, Courtney T. Hill

STEMPS Theses & Dissertations

Calibration, or students’ ability to predict and postdict their performance on an academic task, has important implications for academic performance. It can influence students' decision making processes concerning learning activities, as well as their levels of motivation and self-regulation (Kolovelonis, 2023). The literature on calibration is clear, often indicating that students are not great at these judgements (Lin et. al, 2001). While this phenomenon is well documented, it is not an easy one to explain. Therefore, a two-phase explanatory-sequential mixed method design was used to examine undergraduate students’ calibration judgements on a college algebra exam. To further contribute to the …

Relationship Between Local Fiscal Effort And High School Graduation Rate, Zachary Michael Haney

Educational Leadership & Workforce Development Theses & Dissertations

Education is an investment in human capital, but the cost to provide this public service is significant. Understanding the return on this investment is essential for informed and effective financial decision-making. While previous school finance studies have examined the relationship between funding and various indicators of student achievement and student attainment, most have focused on spending practices rather than the level of investment itself. Although the literature affirms the need for educational resources, further research is required to determine which types of investment most positively affect student outcomes. Therefore, funding must be analyzed prior to budgetary allocation to evaluate its …

Analysis Of Industry Recognized Credentials That Lead To Quality Employment For High School Graduates, Teri L. Harris

Educational Leadership & Workforce Development Theses & Dissertations

The purpose of this study was to identify to what extent industry recognized credentials (IRCs) earned in secondary school lead to high quality employment for students after high school and which specific IRCs should be integrated into career and technical education (CTE) courses. This author sought to explore if CTE courses in high school could be the vehicle for students to earn IRCs that led to higher quality jobs, higher pay, and a job to support a family. The researcher gathered empirical evidence using existing data from a survey of business leaders in each of the nationally recognized 14 Career …

Influence Of Faculty-Derived Concept Maps On Student Study Strategies, Kaitlin P. Hardy, Amanda K. Burbage, Julie A. Kerry

EVMS School of Health Professions Faculty Publications

Background

Concept mapping is a well-established tool for students to actively organize information into a visual and spatial framework. However, some students are resistant to adopting concept mapping as a study strategy due to the time and effort involved in generating the maps. In this study, we explored student perceptions of faculty-derived concept maps (FD-maps) developed to accompany content within a Foundational Science module in a medical school curriculum. We subsequently investigated whether student use of FD-maps helped overcome resistance to adoption of concept mapping.

Methods

Student evaluation data was analyzed using a mixed methods approach to explore student perceptions …

Am I As Effective At Identifying Emotions As Artificial Intelligence? A Comparative Study Of Emotional Recognition, Traci R. Grove, Alexandra T. Lucas, Maryann Martin, Cathleen M. Deckers, Lulu Sherif Mahmood, Nicole Danaher-Garcia, Mark W. Scerbo, Suzan Kardong-Edgren, Janice C. Palaganas

Psychology Faculty Publications

Background

Learning conversations, or dialogues aimed at deepening understanding and reflection, are deeply influenced by emotions. Effective communication is influenced by emotional intelligence - the ability to recognize, understand, and manage both one’s own and others’ emotions. While advances in artificial intelligence (AI) offer new tools for emotion recognition, these technologies still struggle with accurately interpreting subtle and culturally diverse emotional expressions, sparking debate about their reliability and effectiveness. This article provides a comparative analysis of human versus AI recognition of emotions during an end-of-course reflective learning conversation.

Methods

Emotions during a structured post-conference debriefing were analyzed and coded by …

A Comparison Of Service Needs For Face-To-Face To Online Programs, Ali Ardalan, Roya K. Ardalan, G. Steven Rhiel

Information Technology & Decision Sciences Faculty Publications

Higher education institutions strive to gain a competitive advantage over similar programs offered by other institutions. A criterion for comparing programs is the quality of service. The definition of quality from the customer's view is "fitness for use," which incorporates the value generated by the product. Each learner may evaluate programs differently from other learners. It is impossible to develop a unique standard for program quality that is defined by a specific set of factors and specific level for each of these factors. This study presents an extensive set of factors that may affect learners' satisfaction. Higher education administrators should …

How Have Teacher Candidates Learned To Use Evidence? A Systematic Review Of Research, J. S. Beck, D. R. Riddle, J. J. Morgan, H. Whitesides, N. Brown

Teaching & Learning Faculty Publications

The purpose of the current manuscript was to provide a systematic review of the literature regarding how preservice teachers learn to use evidence to evaluate student learning. To that end, we operationalize the terms data literacy for teachers, data-driven decision making, and assessment literacy before outlining the methods of our systematic review. Our review produced 36 relevant articles situated within the United States. Based on these articles, we describe how teacher candidates learn to use data including relevant interventions and instructional methods. We also expand on the types of research methods used in these studies. We conclude with implications for …

Understanding Teacher Leadership: Development And Testing Of The Survey Of Teacher Leadership, Peter D. Wiens, Vanessa Vongkulluksn, Jori Beck, Brenda Pearson

Teaching & Learning Faculty Publications

Teacher leadership (TL) has been identified as key component of effective schools. While research has grown in this area, there is a dearth of empirically and theoretically supported measures for understanding TL. This study examines the validity and reliability of the Survey of Teacher Leadership (STL). The STL builds on previous theory and research in TL by creating an assessment of teacher TL practices and their associated self-efficacy in these practices. In addition to more established domains of TL, the STL also examines teachers’ leadership practices and self-efficacy in diversity and equity. The STL was sent to all non-charter public …

Culture Wars In The Commonwealth: Public Libraries, Book Bans, And The First Amendment, Natisha Harper, Amber Matthews

STEMPS Faculty Publications

Public libraries are quickly losing the ‘culture war’ being waged against them (Jones, 2024). They also lack the coordinated support, resources, and legislative power of those leading the national assault on the right to read, intellectual freedom, and the First Amendment (ALA, 2024). Few resources provide timely and relevant guidance for public libraries on negotiating book challenges with concerned community members, elected officials, and interested parties. This research study responds to nascent ‘culture war’ developments in Virginia – specifically, recent government interference in public library governance in Warren County in December (Schneider, 2024) and a new bill targeting ‘obscene’ materials …

Application Of Fuzzy Logic For Evaluation Of Student Performance, M. Tomovic, C. Tomovic, V. Jovanovic, S. Bawab

Engineering Technology Faculty Publications

The evaluation of student performance is a multifaceted process that traditionally relies on rigid grading systems, often failing to capture the complexity of student abilities and learning dynamics. This paper explores the application of fuzzy logic to enhance the accuracy and adaptability of student performance evaluation. By leveraging fuzzy logic principles, such as membership functions and linguistic variables, we model and assess the various factors influencing student performance—ranging from academic grades to engagement, participation, and personal learning styles. The approach allows for the representation of imprecise, uncertain, and subjective data, offering a more holistic view of a student’s progress. We …

The Security Of Deep Neural Networks, Jalaya Allen

Cybersecurity Undergraduate Research Showcase

Our society has transitioned from our primitive lifestyle to soon, an increasingly automatic one. That idea is further exemplified as we shift into an AI era, better known as Artificial intelligence. Artificial Intelligence is classified as computer systems that can perform tasks that typically require human intelligence. However, a common thought or question that most might have is, how is this done? How does AI process information the way we want it to and have access to so much information? AI is trained by systems called AI models. These modeling programs are trained on data to recognize patterns or make …

The Impact Of Guided Notes Intervention On Learning Achievement, Quality Of Notes, And Perception, Bryan Richard Biggers

STEMPS Theses & Dissertations

Guided notes in the form of teacher-prepared materials that guide students through the lesson with cues and space have been recommended as a possible aid to support student learning. The present study investigated the results of guided notes on the students’ learning achievement, the quality of the students’ notes, and the students’ perceptions of note-taking. Sixty students were enrolled in an eight-week, asynchronous, online, general education course and were divided into nonintervention and intervention groups where the nonintervention groups took notes without support while the students in the intervention group took notes with guided notes during weeks two through seven. …

Global Strategies For Addressing Communication Disorders In Justice-Involved Youth, Sophia Martinez, Anne Marie Perrotti, Brooke Baker, Allison Chappell

Undergraduate Research Symposium

Throughout all stages of processing, most juveniles who encounter the justice system demonstrate cognitive-communicative difficulties (CCDs). Empirical evidence reveals that 19-22% of female and 28-38% of male youth offenders, culminating in approximately 50-60% of youth offenders, demonstrate language disorders that require intervention, but most have never received services (Snow et al., 2014; Winstanley et al., 2021). Speech-language pathologists (SLPs) are communication experts certified to assess and treat CCDs. While nations abroad recognize the value of SLP involvement in all stages of processing for justice-involved youth (JIY), America lags in utilizing SLPs to improve JIY outcomes and lower youth's risk of …

The Song Of Life: Listening To Self, Others, Nature, And God (The Divine), E. James Baesler

Communication & Theatre Arts Faculty Publications

“Listening to the SONG of Life” is a pedagogical framework that conceptualizes listening as a multi-sensory experience in four contexts: Self, Others, Nature, and God/Goddess (the Divine). L-SONG (Listening-SONG) is an instrument developed to measure student learning in four contexts during a listening course.

Multifaceted Challenges And Opportunities: Concurrent Mixed Methods Research To Investigate Chinese Exchange Students' Experiences In The Us Transnational Online Learning Ecology, Xinyue Ren, Yi Zhou

STEMPS Faculty Publications

Transnational online education has become an emerging trend in promoting academic collaboration and exchange in higher education during and after the pandemic. Guided by the learning ecology framework, we conducted concurrent mixed methods research by distributing an online survey to Chinese students (N=51) to collect quantitative and qualitative data. This helped us deeply understand their experiences and perceptions of a cross-cultural online learning environment in the United States. We applied multiple regression and thematic analysis to analyze the data. The merging of both types of data analysis indicated the confirmation of the importance of discipline-based English learning environments, the transformation …

A Systematic Review Of Educational Online Peer-Review And Assessment Systems: Charting The Landscape, Dmytro Babik, Edward Gehringer, Jennifer Kidd, Kristine Sunday, David Tinapple, Steven Gilbert

Teaching & Learning Faculty Publications

Over the past two decades, there has been an explosion of innovation in software tools that encapsulate and expand the capabilities of the widely used student peer assessment. While the affordances and pedagogical impacts of traditional in-person, "paper-and-pencil" peer assessment have been studied extensively and are relatively well understood, computerized (online) peer assessment introduced not only shifts in scalability and efficiency, but also entirely new capabilities and forms of social learning interactions, instructor leverage, and distributed cognition, that still need to be researched and systematized. Despite the ample research on traditional peer assessment and evidence of its efficacy, common vocabulary …

Identifying Support Structures Associated With Informal Formative Evaluation In Instructional Design, Suzanne Smith, Tian Luo

STEMPS Faculty Publications

Instructional designers perform informal formative evaluation in design practice. An evaluation may be used to locate errors in alignment of instructional objectives or to increase the quality or effectiveness of a design. An instructional design review is similar to peer reviews in higher education which are often structured, and tools are provided to contribute to the review. A study was performed to identify the support structures and tools that contribute to building a community of feedback within the practice of instructional design reviews. Six instructional designers and design managers were interviewed to gather the processes they use in design reviews …

Caregivers' Perceptions Of Covid-19 Educational Disruptions On Children With Developmental Language Disorder And Typically Developing Peers, Katherine Radville, Danika Pfeiffer, Karynn Sheranian, Julie Wolter, Jessie Ricketts, Tiffany L. Hogan

Communication Disorders & Special Education Faculty Publications

Purpose: Understanding the experiences of families of children with developmental language disorder (DLD) during COVID-19 educational disruptions is essential for designing responsive supports during pandemic recovery efforts and beyond. This qualitative study describes the experiences of families of first-and second-grade children with DLD during the pandemic as compared to the experiences of families of typically developing (TD) peers.

Method: A conventional content analysis approach was used to analyze caregivers’ written responses to open-ended questions regarding their perceptions of COVID-19 educational disruptions. Responses were analyzed separately by group: caregivers of children with DLD (n= 23) and caregivers of TD children (n= …

Learner Assessment, Elizabeth Burns

STEMPS Faculty Publications

The article discusses the importance of learner assessment in school libraries, highlighting the need for school librarians to incorporate assessment practices aligned with the National School Library Standards. Assessment in school libraries focuses on measuring competencies aligned with real-world information-seeking behaviors rather than traditional grades and testing. The article emphasizes the role of diagnostic, formative, and summative assessments in tracking learner progress and supporting the overall library program. It also underscores the significance of using learner data to establish library goals and showcase the impact of school libraries on academic achievement.

The Underrepresentation Of Black Females In Cybersecurity, Makendra Latrice Crosby

Cybersecurity Undergraduate Research Showcase

The significance of cybersecurity methods, strategies, and programs in protecting computers and electronic devices is crucial throughout the technological infrastructure. Despite the considerable growth in the cybersecurity field and its expansive workforce, there exists a notable underrepresentation, specifically among Black/African American females. This study examines the barriers hindering the inclusion of Black women in the cybersecurity workforce such as socioeconomic factors, limited educational access, biases, and workplace culture. The urgency of addressing these challenges calls for solutions such as education programs, mentorship initiatives, creating inclusive workplace environments, and promoting advocacy and increased awareness within the cybersecurity field. Additionally, this paper …

Ways To Make Cybersecurity Education/Opportunities More Accessible In The Philippine Public School System, Joshua Oania

Cybersecurity Undergraduate Research Showcase

This paper will examine how the Philippines can make cybersecurity education more accessible in their public school system. The solutions it proposes include making cybersecurity a part of the school curriculum, creating summer/internship programs for Junior and Senior High School students in multiple different areas within cybersecurity, and providing basic infrastructure and resources for students to meet their educational needs and aspirations.

Relationship Between Learning Communities And Campus Residency On Sense Of Belonging Among First Year Undergraduates, Lanah Kathryn Stafford

Educational Leadership & Workforce Development Theses & Dissertations

In higher education, sense of belonging has been linked to improved outcomes such as retention and academic achievement (e,g., Pittman & Richmond, 2007). Previous research has also identified participation in learning communities and campus residency as interventions which improve sense of belonging for students in this context (e.g., Solanki et al., 2019; Soria & Roberts, 2021). However, little research has been conducted to empirically study sense of belonging as a multidimensional construct, and no research has investigated interactions between on-campus residency and learning community participation on its development.

Data were collected from 1,072 first-year students on the Sense of Belonging …

Evaluating The Effectiveness Of Recent Educational Foundations Requirements By The Commission On Accreditation In Physical Therapy Education For Program Directors Of Physical Therapist Assistant Programs, Sharan Dawn Zirges

Educational Leadership & Workforce Development Theses & Dissertations

On November 2015, the Commission on Accreditation of Physical Therapy Education (CAPTE) published position papers for the 2016 changes to the CAPTE standards. One of these position papers discussed the changes in the 2016 standards related to the qualifications for the position of a program director (PD) for a physical therapist assistant. A specific qualification requirement was added stating “experience in educational theory and methodology, instructional design, student evaluation and outcome assessment, including the equivalent of nine academic (semester) credits of coursework in educational foundations.” The effectiveness of this additional requirement has to date not been assessed. The purpose of …