Digital Commons Network^™

The Effects Of Rater Training On Rater Effects And Validity Of Direct Behavior Ratings, Abigail E. Pruitt

Graduate Research Theses & Dissertations

Educators have a responsibility to accurately measure student behavior in order to identify students in need of additional behavioral support. Current behavior screening tools can be lengthy or difficult to complete, and Direct Behavior Ratings (DBRs) offer a solution. However, DBRs are rater-mediated assessments, prone to rater effects. Rater training methods can be used to mitigate these rater effects, but previous research has not investigated the best training method for reduction of rater effects. Additionally, Many Facet Rasch Measurement (MFRM) provides an opportunity to adjust student ratings in response to individual rater tendencies of severity and leniency. Therefore, the primary …

Construct Validity Of The Behavior Assessment System For Children-Third Edition Teacher Rating Scales (Basc-3 Trs): Comparisons With The Adjustment Scales For Children And Adolescents (Asca), Shannon Burback

Masters Theses

The Behavior Assessment Scale for Children-Third Edition Teacher Rating Scale Child Form (BASC-3 TRS-C) and the Adjustment Scales for Children and Adolescents (ASCA) are both teacher rating scales which may be used by school psychologist to assess youth behavior problems. The BASC, BASC-2, and BASC-3 have limited replicated research of the studies reported in their respective manuals. Therefore, it was important to empirically compare the BASC-3 TRS-C with the ASCA to examine construct validity (convergent, discriminant, and divergent) as there were, at present, no published studies replicating BASC-3 Manual research. The present study analyzed BASC-3 TRS-C and ACSA ratings which …

Measuring Cogency In Argument In The Seventh-Grade English Classroom, Millie Gonzalez-Balsam

Doctoral Dissertations

Constructing a cogent argument that addresses real-world problems aids students in the development of critical thinking and requires students to present multiple perspectives in a credible manner. Yet, rubrics do not always measure students’ reasoning. The purpose of this study was to create a valid and reliable instrument to measure cogency in argument. I created a Teacher Designed Rubric Measuring Cogency (TDRMC) based on Toulmin’s model of argument for its emphasis on context-specific warrants, and I used Wilson’s framework for assessment to operationalize the construct of cogency. I compared the TDRMC to the current standardized assessment rubric for the Common …

The Concurrent Validity Of The Learning Component Of The Missouri Ability Scale, Nicholas Johnson

Murray State Theses and Dissertations

ABSTRACT

The present study was designed to determine the concurrent validity of the Missouri Ability Scale (MAS), a new measure of independent functioning and learning currently in development. The MAS consists of 10 subtests and is designed to be administered to the examinee and an informant. Fifty individuals (M = 13.1 years; SD = 5.8 years) were administered the MAS and a cognitive abilities test (i.e. WISC-V, KABC-II, WJ-IV). Overall, the Spearman correlations between the MAS learning component and the measures of intellectual ability were moderate-to-strong, indicating good validity. Consistent with the hypotheses, the MAS learning component and the Cattel-Horn-Carroll …

School Culture And Climate For Younger Learners: Measurement And Association With Academic Achievement, Leon Joseph Gilman

Theses and Dissertations

This study seeks to understand the measurement of younger students’ perceptions of the school learning environment and their possible association with academic achievement. The target population is 4th and 5th grade students. Their perception of the school environment was compared to 7th graders by factor analysis, measurement invariance, differential item functioning, and hierarchical linear modeling. This study found that younger students’ perceptions are different from middle school students. However, like their middle school peers, these perceptions still predict academic performance.

Extending The Model With Internal Restrictions On Item Difficulty (Mirid) To Study Differential Item Functioning, Yong "Isaac" Li

USF Tampa Graduate Theses and Dissertations

Differential item functioning (DIF) is a psychometric issue routinely considered in educational and psychological assessment. However, it has not been studied in the context of a recently developed componential statistical model, the model with internal restrictions on item difficulty (MIRID; Butter, De Boeck, & Verhelst, 1998). Because the MIRID requires test questions measuring either single or multiple cognitive processes, it creates a complex environment for which traditional DIF methods may be inappropriate. This dissertation sought to extend the MIRID framework to detect DIF at the item-group level and the individual-item level. Such a model-based approach can increase the interpretability of …

Reliability And Validity Of Michigan School Libraries For The 21st Century Measurement Benchmarks, Natosha Nicole Floyd

Wayne State University Dissertations

The purpose of this study was to examine the psychometric properties of the Michigan School Libraries for the 21st Century Measurement Benchmarks (SL21). The instrument consists of 19 items with three subscales: Building the 21st Century Learning Environment Subscale, Teaching for 21st Century Learning Subscale, and Leading the Way to 21st Century Learning Subscale. The sample consisted of 54 respondents who were administered the instrument in 2014 and 2015. Cronbach’s alpha for the total instrument was 0.807 (n = 19 items). Exploratory factor analysis (EFA) was used to measure construct validity. The findings derived from the EFA did not tend …

Exploring Validity Of Computer-Based Test Scores With Examinees' Response Behaviors And Response Times, Fusun Sahin

Legacy Theses & Dissertations (2009 - 2024)

Examining the testing processes, as well as the scores, is needed for a complete understanding of validity and fairness of computer-based assessments. Examinees’ rapid-guessing and insufficient familiarity with computers have been found to be major issues that

Validation Study Of The Science Literacy Assessment: A Measure To Assess Middle School Students' Attitudes Toward Science And Ability To Think Scientifically, Tammy Mckeown

Theses and Dissertations

This study investigated validity evidence for the Science Literacy Assessment, an instrument designed to assess middle school students’ ability to think scientifically as well as their motivation and beliefs about science (Fives, Huebner, Birnbaum, & Nicolich, 2014). Specifically, three sources of evidence were considered; internal structure, concurrent criterion-related, and predictive criterion-related. Exploratory factor analysis was utilized to examine the underlying factor structure of each of the instrument’s two components, motivation and beliefs related to science and demonstrated scientific literacy. Pearson product-moment correlations were calculated to determine the relationship between scores on the motivation and belief component of the Science Literacy …

Evidence For The Validity Of The Student Risk Screening Scale In Middle School: A Multilevel Confirmatory Factor Analysis, Matthew Porter Wilcox

Theses and Dissertations

The Student Risk Screening Scale—Internalizing/Externalizing (SRSS-IE) was developed to screen elementary-aged students for Emotional and Behavioral Disorders (EBD). Its use has been extended to middle schools with little evidence that it measures the same constructs as in elementary schools. Scores of a middle school population from the SRSS-IE are analyzed with Multilevel Confirmatory Factor Analysis (MCFA) to examine its factor structure, factorial invariance between females and males, and its reliability. Several MCFA models are specified, and compared, with two retained for further analysis. The first model is a single-level model with chi-square and standard errors adjusted for the clustered nature …

Evaluating The Validity Of Technology-Enhanced Educational Assessment Items And Tasks: An Empirical Approach To Studying Item Features And Scoring Rubrics., Ally Thomas

Dissertations, Theses, and Capstone Projects

With the advent of the newly developed Common Core State Standards and the Next Generation Science Standards, innovative assessments, including technology-enhanced items and tasks, will be needed to meet the challenges of developing valid and reliable assessments in a world of computer-based testing. In a recent critique of the next generation assessments in math (i.e., Smarter Balanced), Rasmussen (2015) observed that many aspects of the technology “enhancements” can be expected to do more harm than good as the computer interfaces may introduce construct irrelevant variance. This paper focused on issues surrounding the design of TEIs and how cognitive load …

A Grounded Theory Exploration Of The North Carolina Educator Evaluation System And Its Effects On Teaching Practices And Teacher Leadership, Daniel A. Wydo

Education Dissertations and Projects

This study examined the effects of the recently implemented North Carolina Educator Evaluator System (NCEES) on teaching practices and teacher leadership in a mostly rural county in the Piedmont region of North Carolina. NCEES is designed to improve teaching practices and teacher leadership through performance-based standards. This mixed-methodology study began using grounded theory to form categories from qualitative data collected from piloted focus groups and interviews. Categories derived from the grounded theory analysis were refined in a secondary research method guided by a historical analysis of the processes related to teacher evaluation systems across many decades. The refined categories were …

A Comparison Of Latent Class Analysis And The Mixture Rasch Model: A Cross-Cultural Comparison Of 8th Grade Mathematics Achievement In The Fourth International Mathematics And Science Study (Timss-2011), Turker Toker

Electronic Theses and Dissertations

This study provides a comparison of the results of latent class analysis (LCA) and mixture Rasch model (MRM) analysis using data from the Trends in International Mathematics and Science Study - 2011 (TIMSS-2011) with a focus on the 8th-grade mathematics section. The research study focuses on the comparison of LCA with Mplus version 7.31 and MRM with WinMira 2011 to determine if results obtained differ when the assumed psychometric model differs. Also, a log-linear analysis was conducted to understand the interactions between latent classes identified by LCA and MRM. The data set used in the study was from four diverse …

Predictive Validity Of Curriculum-Based Reading Measures For High-Stakes Outcome Assessments With Secondary Students Identified As Struggling Readers, Tierney Gifford

Legacy Theses & Dissertations (2009 - 2024)

Curriculum-based measurement (CBM) tools are used widely to assess students’ progress within different stages of the Response to Intervention (RTI) process. Despite the wide-spread use, little research has identified the efficacy of reading CBMs in predicting secondary student outcomes on high-stakes assessments. High-stakes assessments are being utilized to determine outcomes for not just students, but teachers, administrators, and districts. More research is needed to determine if reading CBMs are useful tools for the populations of struggling secondary readers. The current study was a secondary analysis of existing data, which attempted to gain an understanding of this through examining the predictive …

An Empirical Comparison Of The Effect Of Missing Data On Type I Error And Statistical Power Of The Likelihood Ratio Test For Differential Item Functioning: An Item Response Theory Approach Using The Graded Response Model, Patricia Rodriguez De Gil

USF Tampa Graduate Theses and Dissertations

In the context of educational research, missing data arise when examinees omit or do not reach an item, which generates an item nonresponse problem. Using a simulation approach, in addition to conducting complete data analyses, this study compared the performance of six methods for treating item nonresponse in the context of differential item functioning (DIF). The effect of missing data on the Type I error and statistical power of the Likelihood Ratio test for DIF detection in small scales was examined in the context of Item Response Theory (IRT-LR), using polytomous, Likert-type data and the graded response model. The effect …

Validity And Rater Reliability Of Peer And Self Assessments For Urban Middle School Students, Lucas Jackson

Theses and Dissertations

This project studied the validity and reliability of self and peer assessments used for group work. The targeted population is middle school students in urban schools. The sample includes 45 sixth graders selected from a public middle school in a large Midwestern metropolitan area. The students worked in groups to complete a classroom project. Self and peer assessment forms were used to rate each member's contribution to the group work. A Generalizability Theory design was used to evaluate the reliability of self and peer assessments. The validity of student ratings was assessed by comparing them to those assigned by the …

The Reliability And Validity Of The Thin Slice Technique: Observational Research On Video Recorded Medical Interactions, Tanina Suzanne Foster

Wayne State University Dissertations

The Reliability and Validity of the Thin Slice Technique: Observational Research on Video Recorded Medical Interactions

Introduction: Observational research using the thin slice technique has been routinely incorporated in observational research methods, however there is limited evidence supporting use of this technique compared to full interaction coding. The purpose of this study was to determine if this technique could be reliability coded, if ratings are consistent between the first, second and third slice, and if they are indeed representative of full interactions.

Methods: Three 30-second thin slices were sampled from the beginning, middle and end of a full-length video-recorded …

Validity Of The Educator Evaluation Instrument In The State Of West Virginia, Carla Howe

Wayne State University Dissertations

In the state of West Virginia, the educator evaluation system was implemented in 2010 as part of a comprehensive system of support to increase teacher effectiveness and student learning. As part of the system, the Educator Evaluation Instrument was developed to measure teachers' effectiveness. This study was conducted to determine whether the Educator Evaluation Instrument was valid for use in measuring effectiveness.

A hierarchical confirmatory factor analysis (HCFA) was conducted on the scores from the demonstration year. The data were not normal, nor was good model fit established based on the current model. Because good model fit could not be …

Value-Added And Observational Measures Used In The Teacher Evaluation Process: A Validation Study, Claudia Güerere

USF Tampa Graduate Theses and Dissertations

Scores from value-added models (VAMs), as used for educational accountability, represent the educational effect teachers have on their students. The use of these scores in teacher evaluations for high-stakes decision making is new for the State of Florida. Validity evidence that supports or questions the use of these scores is critically needed. This research, using data from 2385 teachers from 104 schools in one school district in Florida, examined the validity of the value-added scores by correlating these scores with scores from an observational rubric used in the teacher evaluation process. The VAM scores also were examined in relation to …

Psychometric Properties Of Postsecondary Students' Course Evaluations, Michael J. Drysdale

All Graduate Theses and Dissertations, Spring 1920 to Summer 2023

Several experts in the area of postsecondary student evaluations of courses have concluded that they are stable or reliable measures as well as being measures that provide ways of making valid inferences regarding teacher effectiveness. Often these experts have offered these conclusions without supporting evidence. Surprisingly, a thorough review of the literature revealed very few reported test-retest reliability studies of course evaluations and the results from these studies are contradictory. In the area of validity, the conclusions offered by scholars who conducted meta-analyses of mutlisection course studies are inconsistent. This leads to the following two research questions:

1. What is …