Educational Assessment, Evaluation, and Research Commons^™

Best Practices For Research Analytics & Business Intelligence Within The Research Domain, Baron Wolf, Terri Hall, Katherine Robershaw

Research Analytics Academic Publications

While an increased number of industries with business operations have been shaping their use of data analytics, the use of such tools and methods within the higher education research domain, specifically research administration, is still in its infancy. This mixed-methods study collected data to identify best practices in how universities and other research organizations use data analytics to drive their strategic agendas, create efficiency in operations, and promote complex research proposals throughout their institutions. Research methods included a survey to collect data on how research offices are using analytics and business intelligence tools, Rasch analysis (Rasch, 1993) to examine survey …

Rate To Measure Mathematics Teaching: Using The Many-Facet Rasch Modeling To Reevaluate The Mathematics Classroom Observation Protocol For Practices (Mcop2), Chunling Niu

Theses and Dissertations--Education Sciences

Rater-mediated classroom observation protocols are increasingly being used for teaching performance assessments, which makes identifying and controlling for various rater effects a central issue to ensure the rating quality. A series of validation studies under the classical test theory framework, including content validity, interrater reliability, and structure analysis, have been completed for the 16-item Mathematics Classroom Observation Protocol for Practices (MCOP²).

However, the MCOP² data have never been investigated under the Rasch framework. Due to the methodological limitations of the CTT approach for rater-mediated assessments, it is imperative to examine the MCOP² validity and reliability using …

The Past And Future Of Teacher Efficacy, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

Knowing how the concept of teacher efficacy has evolved in education research shows us good ways to raise teachers' confidence now.

Learning From Failures: Lessons From Unsuccessful Grading Reform Initiatives, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

This article describes accounts of grading reform initiatives that while well-intentioned, met with staunch opposition and eventually were abandoned. The implementation strategies employed by the leaders of these reform initiative are explored, along with reasons these strategies failed to result in meaningful and enduring change. Alternative grading reform strategies with supporting evidence are offered, justification for their use explained, and new directions for grading reform initiatives recommended.

The “Case” For Assessments: An Evaluation Of Case End Of Course Assessments And Teacher Motivation, Andrew Lawrence Hauser

Theses and Dissertations--Community & Leadership Development

Curriculum for Agricultural Science Education (CASE) End of Course (EoC) assessments are summative assessments used to evaluate the CASE curriculum. This quantitative study explored the effects teacher motivation has on student performance on CASE EoC assessments. CASE teachers (n= 55) who participated in the study responded to the 28-item CASE End of Course Assessment Teacher Motivation Questionnaire on a 5-point Likert-type scale. The researcher conducted a Pearson r Correlation test to determine a relationship between questionnaire mean values and secondary student assessment data. Results from the study outline frequency of responses, perceived value and expectancy mean values, and correlations …

Is What Works Working? Thinking Evaluatively About The What Works Clearinghouse, Andrew A. Nelson

Theses and Dissertations--Education Sciences

Since the mid-twentieth century, the U.S. Department of Education has drafted and enacted policies to bridge the research-practice gap—that is, the gap between “what works” according to educational research and what is actually practiced by teachers and their administrators (e.g., Dirkx, 2006; Joyce & Cartwright, 2019; Tseng, 2012). One of the latest manifestations of this “what works” political legacy is the What Works Clearinghouse (WWC), which took shape as part of the Institute of Education Sciences (IES) in 2002. The WWC’s mission is to be a “central and trusted source of scientific evidence for what works in education” (WWC, 2020d, …

Important Attributes Of A Behaviorally-Based Coaching Culture, Sarah M. Eustice

Theses and Dissertations--Education Sciences

Professional coaching is still considered an emerging field. Over the past 30 years, professional coaches and researchers have worked to further this profession by developing coaching competencies, exploring the nuances between various coaching modalities, and developing models to explain the role of coaching in organizations. There remains, however, a dearth of literature on coaching culture, including limited agreement on what it is and how it should look in an organization.

This study aims to establish agreement by investigating the confluence of coaching culture definitions and models present in industry and scholarly literature, and garnering input from a sample of professionals …

Teacher Perceptions Of School Climate: A New Instrument And Validity Study, Rui Jin

Theses and Dissertations--Education Sciences

The United States government has growingly emphasized school climate, as it has been involved as an accountability indicator in the Every Student Succeeds Act. However, there remains considerable debate on both conceptualizing and measuring school climate. Moreover, teachers, a critical informant of school climate, have been ignored because most previous studies have paid attention to the students.

To better inform the research and practice related to the teacher perceptions of school climate and its measurement, this dissertation first identified the related instruments in the literature. The results showed only one survey measured three domains of school climate (academic climate, community, …

Exploring The Impact Of Pre-College Stem Exposure On First-Year Engineering Student Self-Efficacy, Micaha Dean Hughes

Theses and Dissertations--Science, Technology, Engineering, and Mathematics (STEM) Education

STEM (science, technology, engineering, and mathematics) outreach programs for high school students have become more widely available during the last decade in an effort to drive more graduates into the ever-growing STEM field in the United States. This study utilizes the Longitudinal Assessment of Engineering Self-Efficacy (LAESE) survey to investigate a possible relationship between specific pre-college STEM outreach programs and noncognitive predictors of success in first-year undergraduate engineering students. Students in their first semester of a first-year engineering program at a large, public research university completed the LAESE survey and were asked to identify their participation in one of three …

The Impact Of Standards-Based Learning: Tracking High School Students’ Transition To The University, Thomas R. Guskey, Matt Townsley, Thomas M. Buckmiller

Educational, School, and Counseling Psychology Faculty Publications

This study sought to determine if the implementation of standards-based learning in high schools affects students’ transition to learning in university courses. Surveys and interviews with 13 students who had graduated from high schools implementing standards-based learning and who had completed their first academic semester at a midsize, private, Midwest university revealed no detrimental effects. The most frequently mentioned transition difficulties related to social issues and time management. Implications for implementing high school grading reforms are discussed.

Chart A Clear Course: Evaluation Is Key To Building Better, More Relevant Learning, Chase Nordengren, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

When we engage in professional learning, we do it for one big reason: to get better at supporting students. Rigorous and thoughtful program evaluations can provide the critical connection between well-designed programs or initiatives and continuous improvement that builds essential knowledge and skills for educators. Evaluation helps us examine what has been accomplished in a professional learning initiative and identify course corrections that can help the initiative improve.

A Digital Ethnography Of Teach For America: Analysis Of Counternarrative From The Truth For America Podcast, Julian Vasquez Heilig, T. Jameson Brewer, Amber K. Kim, Miguel Sanchez

Educational Policy Studies and Evaluation Faculty Publications

To analyze the counternarrative in the public discourse surrounding Teach For America (TFA), this paper represents the first digital ethnography in education policy. We conduct a qualitative analysis of Truth For America, an education policy podcast. We found four overarching themes that arose from conversations with respondents: (1) problematic practice, preparation, and pedagogy; (2) concerns linked to critiquing TFA and the organization’s responses to that critique; (3) issues related to race and diversity; and (4) disconcerting funding practices and political power. We conclude by discussing the implications of how individual-level stakeholder experiences inform the public discourse about TFA.

Breaking Up The Grade, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

To make grading more meaningful, course grades should reflect a range of distinct criteria that make up student learning.

Eight Essential Principles For Improving Grading, Susan Brookhart, Thomas R. Guskey, Jay Mctighe, Dylan Wiliam

Educational, School, and Counseling Psychology Faculty Publications

Done well, grading can play a key role in a balanced district assessment system.

Applying The Rasch Model To Evaluate The Self-Directed Online Learning Scale (Sdols) For Graduate Students, Hongwei Yang, Jian Su, Kelly D. Bradley

Educational Policy Studies and Evaluation Faculty Publications

With the rapid growth of online learning and the increased attention paid to student attrition in online programs, much research has been aimed at studying the effectiveness of online education to improve students’ online learning experience and student retention. Utilizing the online learning literature as a multi-faceted theoretical framework, the study developed and employed a new survey instrument. The Self-Directed Online Learning Scale (SDOLS) was used to examine graduate student perceptions of effectiveness of online learning environments as demonstrated by their ability to take charge of their own learning, and to identify key factors in instructional design for effective improvements. …

What Do Grades Mean? Variation In Grading Criteria In American College And University Courses, Anastasiya A. Lipnevich, Thomas R. Guskey, Dana M. Murano, Jeffrey K. Smith

Educational, School, and Counseling Psychology Faculty Publications

This study examined differences in the criteria used by college and university instructors in the United States to assign course grades. Two hundred and fifty course syllabi (159 from universities and 91 from four-year colleges) developed by randomly selected instructors from five academic disciplines (education, maths, science, psychology, and English) were examined to determine the extent to which instructors employed different criteria in assigning course grades in introductory-level courses. Sources of variation in grade assignment included the use of product versus process criteria, the prevalence of using performance exams, and the framing criteria for grades. Differences between institution types and …

Policy Storms At The Central Office: Conflicting Narratives Of Racial Equity And Segregation At School Committee Meetings, Serena M. Wilcox

Educational Policy Studies and Evaluation Faculty Publications

This article reports findings from a multiyear critical ethnography that examined race talk dilemmas of school leaders at the central office at a small urban school district to understand why racialized educational policies and practices still persist against African American students. This study takes a structural approach to investigating the impact that race talk has on educational policymaking at the local district level. The guiding research question in this paper examines how we can understand educational reform and policy implementation and the unintended consequences of those interventions through the local from a historical context.

The Dark Side Of Assessment Literacy: Avoiding The Perils Of Accountability, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

Educational measurement and evaluation experts generally agree that increasing stakeholders’ assessment literacy will yield a variety of positive benefits, especially broadening the range of assessment formats teachers use to measure students’ mastery of high level, more cognitively complex learning outcomes. But in the context of education accountability as currently structured in American schools, such efforts also may lead teachers to become more sophisticated in test preparation activities and to narrow both their instruction and classroom assessment practices specifically to enhance students’ performance on prescribed, annual high-stakes accountability assessments. This article explains why that is so, describes the process by which …

When Your Plate Is Already Full: Efficient And Meaningful Outcomes Assessment For Busy Law Schools, Melissa N. Henke

Law Faculty Scholarly Articles

The American Bar Association (ABA) accreditation standards involving outcome-based assessment are a game changer for legal education. The standards reaffirm the importance of providing students with formative feedback throughout their course of study to assess and improve student learning. The standards also require law schools to evaluate their effectiveness, and to do so from the perspective of student performance within the institution’s program of study. The relevant question is no longer what are law schools teaching their students, but instead, what are students learning from law schools in terms of the knowledge, skills, and values that are essential for those …

Alternative Thesis Project For Spring 2020 Semester, Elaine Murner

Theses and Dissertations--Early Childhood, Special Education, and Counselor Education

During the Spring 2020 semester, students within the special education program were conducting applied thesis projects within typical contexts as part of their fulfillment of the requirements of a master’s degree program. Due to the coronavirus disease 2019 (COVID-19), public schools and related facilities closed with no plans to reopen within the time frame to allow for graduation for students in the last semester of their graduate program. Students were allowed to complete an alternative thesis assignment in the form of responding to writing prompts followed by an oral defense of the written products, along with questions related to their …

A Bifactor Approach To Dimensionality Assessment, David Michael Dueber

Theses and Dissertations--Education Sciences

Bifactor confirmatory factor analysis models and statistical indices computed from them have previously been used to provide evidence for the appropriateness of utilizing a unidimensional interpretation of multidimensional data. However, the ability of bifactor indices to aid in the assessment of subscore strength has not been investigated.

A simulation study was conducted to relate bifactor indices to the strength of subscores corresponding to specific factors. The bifactor indices OmegaHS and ECV_SS were found to be strongly predictive of subscore strength conditional upon OmegaS. The number of factors was also found to play a minor role in this relationship. Cutoffs …

Encouraging Healthy Eating Among Older Adults Using The Transtheoretical Model: An Evaluation Of A Pilot Intervention, Lauren Brinkman Roberson

Theses and Dissertations--Communication

Older adults, defined as those age 60 and above, are at an increased risk for many health-related complications that are directly related to nutrition (Centers for Disease Control and Prevention, 2015). This study highlights the lack of nutrition education material developed for older adults in Kentucky. Such material has great potential to influence the health of older adults (Chernoff, 2001). This study evaluated an intervention developed, by means of formative research, to teach older adults nutrition basics. Both direct and indirect measures related to stages of change for healthy eating behaviors were collected six weeks pre-intervention and then immediately post-intervention. …

A Comparison Of Multiple Baseline Family Of Designs, An Analysis Of A Multiple Probe Study, And Coaching For Practitioners, Katherine Jordan

Theses and Dissertations--Early Childhood, Special Education, and Counselor Education

Overview of Alternative Thesis Project

During the Spring 2020 semester, students within the applied behavior analysis program were conducting applied thesis projects within typical contexts as part of their fulfillment of the requirements of a master’s degree program. Due to the coronavirus disease 2019 (COVID-19), public schools and related facilities closed with no plans to reopen within the time frame to allow for graduation for students in the last semester of their graduate program. Students were allowed to complete an alternative thesis assignment in the form of responding to writing prompts followed by an oral defense of the written products, …

Assessing The Performance Of Two Procedures For Detecting Differential Item Functioning Within The Multilevel Partial Credit Model, Carol Hanley

Theses and Dissertations--Education Sciences

This is a simulation study that evaluates the performances of two models for the detection of uniform differential item functioning (DIF). Simulated data are generated by a multilevel partial credit model (MLPCM). The purpose of this study was to compare the accuracy of two DIF detection procedures, hierarchical ordinal logistic regression (HOLR) for multilevel data and multilevel generalized Mantel-Haenszel (MGMH: French & Finch, 2013; French, Finch, & Imekus, 2019). Conditions manipulated were the number of participants per cluster (20, 40), number of clusters (50, 100, 200), DIF magnitude (0, .4, .8), and magnitude of intraclass correlation coefficient (.05, .25, .45). …

Sexual Health Education In Kentucky's Public High Schools, Marjorie Stanek

Theses and Dissertations--Education Sciences

This dissertation seeks to enhance the understanding of sexual health education efforts, as they currently exist in Kentucky’s public high schools. It will include an examination of the factors that influence the provision of sexual health education in Kentucky schools, including relevant legislation, directives from the Kentucky Department of Education, and the presence of site-based councils within the Commonwealth. Additionally, the suggested curriculum elements from the Kentucky Department of Education’s Academic Standards, which were implemented during the 2019-2020 school year, will be compared to those established within the National Sexuality Education Standards (NSES), which serve as a gold standard …

Understanding Education Abroad With Advanced Quantitative Methodologies: Student Profiles And Academic Outcomes, Jie Dai

Theses and Dissertations--Education Sciences

This three-study dissertation contributes to the research in the field of participation in education abroad, particularly as it relates to student profiles and academic outcomes. Through employing more robust methodologies across the three studies, this dissertation aims not only to understand what are the factors associated with education abroad participation and how these factors interplay with each other, but also to provide a less biased picture of the impact of participation in education abroad on postsecondary educational outcomes. The studies have implications for equitable and inclusive access to education abroad.

The first study begins with the question: who studies abroad? …

Can Less Be More? Study Load And Its Effect On Science Achievement Among Chinese Eighth-Grade Students, Xian Wu

Theses and Dissertations--Education Sciences

The present study addressed a core research question of “Can less be more?” concerning study load, which has become increasingly worrisome and controversial around the world. This idea reflects the philosophy of pursuing efficiency and effectiveness on learning, namely that a lighter study load may result in more successful academic achievement. The study adopted the IPO (input-process-output) model to address five interrelated research questions: 1) What are the structural characteristics of study load? 2) Is there any individual difference in study load? 3) What are the characteristics of teachers and schools under which students tend to have a heavier study …

Muslim College Students And Civic Engagement, Aus Abdulwahhab

Theses and Dissertations--Educational Policy Studies and Evaluation

This study explores the nature of civic engagement of Muslim college students in the United States. The Community of Practice Theory is used as a conceptual structure to explore the relation between meaning, practice, identity and organization for Muslim students active in civic engagement through campus-based Muslim Student organizations. Pluralism, as shared values that combine different people and advocates wider civic participation, was adopted as part of the conceptual framework to see how the Muslim students think, serve, learn, and develop throughout the process of engagement on campus and in the local community. Particular attention is paid to the impacts …

Engaging Patients To Prevent Harm: Implementation Of Hospital Acquired Pressure Injury Prevention Education Videos, Sarah Buschmann

DNP Projects

PURPOSE: The purpose of this proposal is to evaluate the introduction of pressure ulcer prevention videos to at risk adult patients in the intensive care unit within the acute care health setting.

METHODS: The study design is a retrospective chart review to analyze the effect of the pressure ulcer prevention videos on the number of pressure ulcers and incidence rate while comparing Jackson/Cubbin scores. As pressure ulcers continue to rise, patient education through the Get Well Network is a required intervention. This project is relevant in reducing pressure ulcers and increasing patient safety. Data collection took place from July 1 …

Nurse Driven Enteral Nutrition Protocol For Mechanically Ventilated Patients, Deena Shewekah

DNP Projects

Introduction: Delayed enteral nutrition in critically ill patients increases the risk of complications and poor outcomes. Current evidence-based guidelines recommend initiation of enteral nutrition within 24-48 hours of intensive care unit (ICU) admission. The quality improvement (QI) project aim was to examine whether enteral nutrition was initiated within 24- 48 hours of mechanical ventilation after implementation of QI intervention in order to determine practice change.

Methods: The QI protocol was introduced to the Cardiovascular ICU (CVICU) at UK Chandler Hospital (UKHC) nurses via an education session. A post-education electronic survey identified the nurse’s understanding of the protocol. To identify nursing …