Educational Assessment, Evaluation, and Research Commons^™

Interpreting Average Effect Sizes: Never A Center Without A Spread, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

School leaders today are making important decisions regarding education innovations based on published average effect sizes, even though few understand exactly how effect sizes are calculated or what they mean. This article explains how average effect sizes are determined in meta-analyses and the importance of including measures of variability with any average effect size. By considering the variation in effect sizes among studies of the same innovation, education leaders can make better decisions about innovations and greatly increase the likelihood of achieving optimal results from implementation.

Grades Versus Comments: Research On Student Feedback, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

Are comments on student work superior to grades? It depends.

How Traditional Grading Contribute To Student Inequities And How To Fix It, Laura J. Link, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

Grades have long been identified by those in the measurement community as prime examples of unreliable measurement (Brookhart, 1994; Stiggins, Frisbie, & Griswold, 1989). What one teacher considers in calculating students’ grades may differ greatly from another teacher (Guskey & Link, 2019; McMillan, 2001; McMillan, Myran, & Workman, 2002). A major factor contributing to the unreliability of grades is teachers’ inclusion of aspects of students’ behavior in the grades they assign. Despite the recommendation of experts to separate behavior from academic achievement in formulating students’ grades, teachers at all grade levels typically include student behavior as a contributing factor in …

Choice Without Inclusion?: Comparing The Intensity Of Racial Segregation In Charters And Public Schools At The Local, State And National Levels, Julian Vasquez Heilig, T. Jameson Brewer, Yohuru Williams

Educational Policy Studies and Evaluation Faculty Publications

We conduct descriptive and inferential analyses of publicly available Common Core of Data (CCD) to examine segregation at the local, state, and national levels. Nationally, we find that higher percentages of charter students of every race attend intensely segregated schools. The highest levels of racial isolation are at the primary level for public and middle level for charters. We find that double segregation by race and class is higher in charter schools. Charters are more likely to be segregated, even when controlling for local ethnoracial demographics. A majority of states have at least half of Blacks and a third of …

Main And Regional Campus Assessments Of Applicants To A Rural Physician Leadership Program: A Generalizability Analysis, Terry D. Stratton, Clarence Kreiter, Carol L. Elam

Behavioral Science Faculty Publications

While the selection of qualified applicants often relies, in part, on scores generated from a medical school pre-admission interview (MSPI), the growth of regional medical campuses (RMCs) – many with specialized rural tracks, programs, or missions – has challenged schools to accommodate a wider range of stakeholder input. This study examines the reliabilities of main (urban) and regional (rural) campus interviewers’ assessments of applicants to a Rural Physician Leadership Program (RPLP) located in the southeastern United States.

Data from RPLP applicants completing MSPIs on two campuses from 2009-2017 (n = 232) were examined in a generalizability analysis. In two separate …

Opportunities For Self-Evaluation Increase Student Calibration In An Introductory Biology Course, Jennifer L. Osterhage, Ellen Usher, Trisha A. Douin, William M. Bailey

Biology Faculty Publications

Accurate self-evaluation is critical for learning. Calibration describes the relationship between learners’ perception of their performance and their actual performance on a task. Here, we describe two studies aimed at assessing and improving student calibration in a first-semester introductory biology course at a 4-year public institution. Study 1 investigated students’ (n = 310) calibration (the difference between estimated and actual exam performance) across one semester. Students were significantly miscalibrated for the first exam: their predicted scores were, on average, significantly higher than their actual scores. The lowest-performing students had the most inaccurate estimates. Calibration improved with each exam. By …

A Multi-Institutional Analysis Of Instructional Beliefs And Practices In Gateway Courses To The Sciences, Joseph J. Ferrare

Educational Policy Studies and Evaluation Faculty Publications

This paper builds on previous studies of instructional practice in science, technology, engineering, and mathematics courses by reporting findings from a study of the relationship between instructors’ beliefs about teaching and learning and their observed classroom practices. Data collection took place across six institutions of higher education and included in-depth interviews with 71 instructors and more than 140 hours of classroom observations using the Teaching Dimensions Observation Protocol. Thematic coding of interviews identified 31 distinct beliefs that instructors held about the ways students best learn introductory concepts and skills in these courses. Cluster analysis of the observation data suggested that …

The Forgotten Element Of Instructional Leadership: Grading, Thomas R. Guskey, Laura J. Link

Educational, School, and Counseling Psychology Faculty Publications

Getting the school team coordinated on grading and reporting policies—and the purpose of grading—is too often overlooked in instructional leadership.

Student Help-Seeking Behaviors And Teacher Instructional Practices: Examining Their Relationship With U.S. Student Mathematics Achievement, Michael C. Osborne

Theses and Dissertations--Curriculum and Instruction

Even though the United States (U.S.) spends, on average, more money per student than most Organisation for Economic Co-operation and Development (OECD) countries, it continues to lag behind its international peers in mathematics achievement. This study, which responded to the call for educational reforms that improve the mathematics achievement of U.S. students, aimed to examine the issue of student help-seeking behaviors and teacher instructional practices as they interact to affect student mathematics achievement. The Programme for International Student Assessment (PISA) defines student help-seeking behaviors as the ways in which students have a propensity to depend on the knowledge and intellect …

Tanglewood Trail Walking And Voucher Program, Kelci Marie Mchugh

Theses and Dissertations--Nutrition and Food Systems

Rural Appalachian populations have greater rates of obesity and chronic diseases attributed to lesser consumption of fruits and vegetables (F/V) and physical inactivity. Barriers to F/V consumption and physical activity include poor access and affordability. To help overcome these barriers, a community-initiated walking and farmers market F/V voucher program was implemented.

Participants were encouraged to walk (1.2 miles roundtrip) to the local farmers market to redeem a $10 voucher which could only be spent on F/V. The 16-week program (June – Sept) included 121 participants. Data was analyzed by creating two groups, low-engagers and high-engagers that differed significantly in the …

From Mean To Quantiles: Rethinking Individual Differences In Mathematics Achievement And Mathematics Self-Efficacy, Jing Yuan

Theses and Dissertations--Educational, School, and Counseling Psychology

The significance of this dissertation research is twofold with both methodological advancement and empirical update. In this dissertation research, quantile regression (QR) was introduced to social sciences researchers as a response to the weaknesses of the traditional mean-based regression often referred to as multiple regression. General advantages of QR includes being more flexible for modeling data with heterogeneous conditional distributions, more robust to outliers, and having richer characterization and description of the data. Results of QR allow researchers to not only describe a general trend of changes in the effects of the independent variables across a continuous distribution of the …

Differential Item Functioning Among English Language Learners On A Large-Scale Mathematics Assessment, Ruixue Liu

Theses and Dissertations--Education Sciences

English language learner (ELL) is a term to describe students who are still acquiring English proficiency. In recent decades, ELLs are a very rapidly growing student group in United States. In school classrooms, ELLs are learning English and their academic subjects simultaneously. It is challenging for them to hear lectures, read textbooks, and complete tests in English despite of their inadequate English language proficiency (Ilich, 2013). As a result, the increasing number of ELLs in public schools has paralleled the increase in ELLs’ low mathematics performance (NCES, 2016).

Due to the popularization of international large-scale assessments in the recent decade, …

Cat In The Classroom: Understanding Instructor Behavior And Student Perceptions Through Communication Accommodation Theory, Terrell Kody Frey

Theses and Dissertations--Communication

Adjusting one’s communication is a fundamental requirement for human interaction (Gasiorek, 2016a). Individuals adapt communication behavior according to the circumstances surrounding the situation, resulting in different patterns and forms of speech relative to spouses, family members, coworkers, or friends. Yet, researchers in instructional communication have not yet substantially applied adjustment as a theoretical lens for understanding instructor-student classroom interactions (Gasiorek & Giles, 2012; Soliz & Giles, 2014; Soliz & Bergquist, 2016). Apart from overlooking this useful theoretical approach, instructional communication scholarship can also be improved by accounting for 1) shifting group identities in higher education that change how instructors and …

Exploring The Factors Teachers Consider In Determining Students’ Grades, Thomas R. Guskey, Laura J. Link

Educational, School, and Counseling Psychology Faculty Publications

The purpose of this study was to investigate the specific factors teachers consider when assigning students’ report card grades. Data were gathered from 943 K-12 teachers from five school districts in a southeastern state in the United States who completed the Teachers’ Grading Practices Survey. Analyses focused on how teachers weigh different factors in determining report card grades, and if these factors and weights differ among teachers who teach at different grade levels and have different amounts of classroom experience. Results revealed statistically significant differences among teachers at different grade levels but no differences associated with teachers’ years of …

Targeting Food Selectivity In Young Children In A Preschool Classroom Using A Multi-Component Treatment Package, Christina Challed Hesley

Theses and Dissertations--Early Childhood, Special Education, and Counselor Education

The purpose of the study was to evaluate the effectiveness of using a video model, graduated exposure (i.e., touch, smell, try, eat), and positive reinforcement to first increase food exploration, and then increase consumption of non-preferred foods in young children that exhibit food selectivity in a school setting. A multiple probe design across behaviors replicated across participants was used to evaluate the effectiveness of a treatment package. The treatment package consisted of a video model of each target behavior (touch, smell, try, eat) and positive reinforcement which included preferred foods and materials. The results indicated that the treatment package was …

Martin School Alumni Survey, Carl Sims

MPA/MPP/MPFM Capstone Projects

The Martin School of Public Policy and Administration is subject to reaccreditation from the Network of Schools of Public Policy, Affairs, and Administration (NASPAA) every seven years. To initiate the reaccreditation process, the Martin School must conduct a programmatic self-study that analyzes the program’s mission, governance, faculty, students, competencies, resources, and communication. The self-study is beneficial not only for the purpose of reaccreditation but also can better inform the school’s strategic planning process

To assist in providing the self-study relevant data, the Martin School can solicit information from its alumni. Alumni data can be used to establish how well the …

Validation Of A School Climate Instrument Using A Rasch Rating Scale Model, Audrey Conway Roberts

Theses and Dissertations--Education Sciences

A new ESSA indicator of school quality and student success provides flexibility to broaden a states’ definition of school and student success. Educational research has found school success is in part determined by a school’s climate and should be considered in improvement/reform strategies (Cohen et al., 2009; Thapa et al., 2013). Yet, school climate research is often difficult and time consuming, and employs a variety of conflicting definitions and dimensions, instruments, and empirical approaches to determining school climate. Given these significant limitations with current measures, the purpose of this study was to validate an instrument measuring school climate based on …

It’S Not The Programs; It’S The People: Building Human Levers Of Retention In Community Colleges, Kyle Barron

Theses and Dissertations--Educational Policy Studies and Evaluation

Student attrition prior to the completion of a credential is an issue that has increasingly demanded the attention of stakeholders in higher education, particularly in the community college sector, in which less than half of all students complete a credential after six years. The costs of student attrition are high and widespread, ranging from the financial costs for institutions and federal and state governments to the personal and monetary costs paid by those students whose personal and professional goals are not achieved. With the ever-increasing focus on accountability for institutions of higher education and the growing movement toward performance-based funding, …

A Partial Simulation Study Of Phantom Effects In Multilevel Analysis Of School Effects: The Case Of School Socioeconomic Composition, Hao Zhou

Theses and Dissertations--Education Sciences

Socioeconomic status (SES) affects students’ academic achievement at different levels of an educational system. However, misspecified Hierarchical Linear Model (HLM) may bias school SES estimation. In this study, a partial simulation study was conducted to examine how misspecified HLM model bias school and student SES estimation.

The result of this study can be summarized by four important points. First, based on partial simulation procedure, phantom effects of school SES and student SES are real. Second, characteristics of phantom effects are generalized. The stronger the correlation between prior science achievement measure and present science achievement measure, the greater the decrease in …

Assessing The Model Fit Of Multidimensional Item Response Theory Models With Polytomous Responses Using Limited-Information Statistics, Caihong Rosina Li

Theses and Dissertations--Education Sciences

Under item response theory, three types of limited information goodness-of-fit test statistics – M₂, M_ord, and C₂ – have been proposed to assess model-data fit when data are sparse. However, the evaluation of the performance of these GOF statistics under multidimensional item response theory (MIRT) models with polytomous data is limited. The current study showed that M₂ and C₂ were well-calibrated under true model conditions and were powerful under misspecified model conditions. M_ord were not well-calibrated when the number of response categories was more than three. RMSEA₂ and RMSEA_C2 are …

The Effect Of Increased Physical Activity On Academic Performance, Lauren Michelle Willis

Theses and Dissertations--Education Sciences

Increased levels of obesity, particularly among American youth, have consistently been cause for concern over the last few decades. Additionally, the amount of time youth spend being active throughout the day has consistently decreased. Physical activity levels among school-aged children in America are effected by any number of reasons, but this study points to the possibility of time spent being physically active during the school day having the greatest effect on a student’s overall level of physical activity. Increased pressures from different entities on local schools to improve student performance on standardized test scores have contributed to a decline in …

Exploring Influences And Using Intentionality To Develop Work-Based Learning Opportunities: A Faculty Perspective, Lauren Mccrary

Theses and Dissertations--Educational Policy Studies and Evaluation

Understanding the practical implications related to experiential learning is complex. It is important to first view experiential learning as an umbrella term under which work-based learning (WBL) is housed. To further particularize, there are different avenues for students to pursue work-based learning opportunities (WBLOs), with internships and cooperative education being two. Although these are not new learning environments, understanding and clearly defining them (to both internal and external bodies) is increasingly necessary. In Kentucky, WBLOs are part of the political discourse on postsecondary education. The Council on Postsecondary Education (CPE) and local economic development agencies have incorporated strengthening and growing …

A Low-Intensity, Hybrid Design Between A "Traditional" And A "Course-Based" Research Experience Yields Positive Outcomes For Science Undergraduate Freshmen And Shows Potential For Large-Scale Application, Thushani Rodrigo-Peiris, Lin Xiang, Vincent M. Cassone

Biology Faculty Publications

Based on positive student outcomes, providing research experiences from early undergraduate years is recommended for science, technology, engineering, and mathematics (STEM) majors. To this end, we designed a novel research experience called the “STEMCats Research Experience” (SRE) for a cohort of 119 second-semester freshmen with diverse college preparatory levels, demographics, and academic majors. The SRE targeted student outcomes of enhancing retention in STEM majors, STEM competency development, and STEM academic performance. It was designed as a hybrid of features from apprenticeship-based traditional undergraduate research experience and course-based undergraduate research experience designs, considering five factors: 1) an authentic research experience, 2) …

A Growth Curve Analysis Of Mandatory Student Athletics Fees, Willis A. Jones, Michael J. Rudolph, Michael Brown

Educational Policy Studies and Evaluation Faculty Publications

This study used growth curve modeling to estimate the growth trajectory of mandatory student athletics fees at public NCAA Division I universities from 2004–2016. We specifically focused on three measures of athletics fees; total athletics fees, athletics fees per FTE, and athletics fees as a percentage of total student costs. We found that in general the growth trajectory of athletics fees was positive, but that the rate of growth has declined over the years. We also found that on average less than 5% of student costs are directly attributable to athletics fees and that the growth trajectory of athletics fees …

Evaluating Preexisting Qualitative Research Data For Secondary Analysis, Victoria Sherif

Educational Leadership Studies Faculty Publications

In this article, I explore the nature of secondary analysis and provide a brief history of the method. Qualitative secondary analysis is a relatively under-used method in education and the social sciences, often due to the lack of easily accessible, relevant, trustworthy, and complete data. I address some of the potentials and limitations that influence its use and explore criteria for assessing the quality and sufficiency of preexisting qualitative research data. Qualitative secondary analysis has important implications for qualitative researchers, students and practitioners interested in generating new knowledge via unobtrusive, reliable, valid, and time/cost effective research through the wider use …

Dental Students' Skills Assessments: Comparisons Of Daily Clinical Grades And Clinical And Laboratory Assessments, Elizangela Bertoli, Kristy Pepper Lawson, Susan S. Bishop

Oral Health Practice Faculty Publications

The aim of this study was to compare the daily clinical grades of third-year dental students during routine clinical activities involving direct and indirect operative procedures to clinical and laboratory assessments. The authors compared students’ daily clinical grades to graded clinical assessments and compared daily clinical grades to laboratory assessments at one U.S. dental school. A total of 50 third-year students (participation rate: 98%) participated in this study during the school year of 2014–15. The study analyzed the students’ daily clinical grades and graded assessments during regular clinical activities and two laboratory examinations. The results found no statistically significant differences …

School Choice In Indianapolis: Effects Of Charter, Magnet, Private, And Traditional Public Schools, Mark Borends, R. Joseph Waddington

Educational Policy Studies and Evaluation Faculty Publications

School choice researchers are often limited to comparing one type of choice with another (e.g., charter schools vs. traditional public schools). One area researchers have not examined is the effects of different school types within the same urban region. We fill this gap by analyzing longitudinal data for students (grades 3–8) in Indianapolis, using student fixed effects models to estimate the impacts of students switching from a traditional public school to a charter, magnet, Catholic, or other private school. We find that students experience no differences in their achievement gains after transferring from a traditional public school to a charter …

Does Pre-Assessment Work?, Thomas R. Guskey

Educational, School, and Counseling Psychology Faculty Publications

Educators must understand the purpose, form, and content of pre-assessments to reap their potential benefits.

Affect, Motivation, And Engagement In The Context Of Mathematics Education: Testing A Dynamic Model Of Interactive Relationships, Shanshan Hu

Theses and Dissertations--Educational, School, and Counseling Psychology

The present study tested the interactive model of affect, motivation, and engagement (Linnenbrink, 2007) in mathematics education with a nationally representative sample. Self-efficacy, self-concept, and anxiety were indicators of pleasant and unpleasant affect. Intrinsic and extrinsic motivation were indicators of mastery and performance approach. Persistence and cognitive activation were indicators of behavioral and cognitive engagement. The 2012 Programme for International Student Assessment (PISA) supplied a sample of 4,978 students from the United States for structural equation modeling. The results indicated that PISA data overall supported the interactive model. Specifically, PISA data completely supported the specification of the relationship between motivation …

A Tale Of Two Contexts: Mathematics Self-Efficacy Development Among Rural And Urban Students, Brianna L. Weidner

Theses and Dissertations--Educational, School, and Counseling Psychology

Self-efficacy, or a belief in one’s ability to complete a task, has been shown to predict student success and persistence. Rural students have a history of lower college enrollment and degree attainment than urban students. However, no studies have compared self-efficacy or its sources across rural and urban groups. The purpose of this study is to examine differences in how rural and urban middle school students develop self-efficacy and self-efficacy for self-regulated learning in the domain of math. Data were collected from 174 rural students and 1743 urban students in grades 6-8 in the southeastern United States. Measurement invariance analyses …