Open Access. Powered by Scholars. Published by Universities.®
- Discipline
-
- Education (17)
- Educational Assessment, Evaluation, and Research (14)
- Educational Psychology (4)
- Higher Education (4)
- Physical Sciences and Mathematics (3)
-
- Statistical Methodology (3)
- Statistics and Probability (3)
- Social Psychology (2)
- Statistical Models (2)
- Arts and Humanities (1)
- Bilingual, Multilingual, and Multicultural Education (1)
- Christianity (1)
- Communication (1)
- Design of Experiments and Sample Surveys (1)
- Educational Methods (1)
- Interpersonal and Small Group Communication (1)
- Linguistics (1)
- Multivariate Analysis (1)
- Other Psychology (1)
- Personality and Social Contexts (1)
- Religion (1)
- Social Statistics (1)
- Sports Studies (1)
- Keyword
-
- Motivation (6)
- Assessment (4)
- Confirmatory factor analysis (4)
- Higher education (4)
- DIF (3)
-
- Higher education assessment (3)
- IRT (3)
- Item response theory (3)
- Propensity score matching (3)
- Simulation study (3)
- Structural equation modeling (3)
- Examinee motivation (2)
- Low-stakes testing (2)
- MIRT (2)
- Measurement (2)
- Measurement invariance (2)
- Missing data (2)
- Mixture modeling (2)
- Model misspecification (2)
- Multidimensional item response theory (2)
- Multiple imputation (2)
- Priors (2)
- Response styles (2)
- Validity (2)
- 3PL (1)
- AAVE (1)
- AI Literacy (1)
- ANOVA (1)
- Ability distribution (1)
- Adaptive testing (1)
- Publication Year
- Publication
- Publication Type
- File Type
Articles 31 - 50 of 50
Full-Text Articles in Quantitative Psychology
Posterior Predictive Model Checking Of Local Misfit For Bayesian Confirmatory Factor Analysis, Chi Hang Au
Posterior Predictive Model Checking Of Local Misfit For Bayesian Confirmatory Factor Analysis, Chi Hang Au
Masters Theses, 2010-2019
Posterior predictive model checks (PPMC) are one Bayesian model-data fit approach. Thus far, PPMC for Confirmatory Factor Analytic applications focused primarily on global fit evaluation, ignoring the nuanced information in local misfit diagnostics. This study developed a PPMC approach for local misfit and applied it to a test-taking motivation scale. If the PPMC approach is effective, fit conclusions derived from the PPMC approach should be congruent with the fit conclusions derived from the Frequentist approach. Number of item-pairs flagged as misfitting and number of disagreements were computed to evaluate congruence. Congruence is achieved if the number of item-pairs flagged as …
Beyond Motivation: Differences In Score Meaning Between Assessment Conditions, Nikole Gregg
Beyond Motivation: Differences In Score Meaning Between Assessment Conditions, Nikole Gregg
Masters Theses, 2010-2019
Written communication is a skill necessary for not only the success of undergraduate students, but for post-graduates in the workplace. Furthermore, according to employers the writing skills of post-graduates tend to be below expectations. Therefore, the assessment of such skills within higher education is in high demand. Written communication assessments tend to be administered in one of two conditions: 1) course embedded and 2) a low-stakes, non-embedded condition. The current study investigated possible construct-irrelevant variance in writing assessment scores by using data from a mid-sized public university in the Mid-Atlantic region of the United States. Specifically, 157 student products were …
The Influence Of Covariate Measurement Error On Treatment Effect Estimates And Numeric Balance Diagnostics Following Several Common Methods Of Propensity Score Matching: A Simulation Study, Heather D. Harris
Dissertations, 2014-2019
In applied intervention studies, researchers frequently aim to make inferences about the impact of a treatment program on participants. However, applied researchers are often faced with threats to the internal validity of their studies, or the extent to which changes in participants’ outcomes can be attributed to the intervention. When researchers are unable to randomly assign study participants to treatment conditions, changes in the intervention outcome might be confounded with systematic differences in participants’ baseline characteristics. Propensity score matching is one technique that allows researchers to account for threats to the internal validity of a study. Specifically, using propensity score …
In Search Of Equality: Developing An Equal Interval Likert Response Scale, Elisabeth M. Spratto
In Search Of Equality: Developing An Equal Interval Likert Response Scale, Elisabeth M. Spratto
Dissertations, 2014-2019
Attitude scales are an important component of educational and psychological research. One consideration when seeking to make valid inferences from attitudinal data is the issue of the degree to which response options can be assumed to have equal intervals. Many response options on attitudinal measures may produce ordinal-level data rather than interval. This poses a problem for the statistical tests that may be used, as many analyses assume interval-level data. It also poses an interpretational issue if the conceptual distance between response options is not the same – for example, if a researcher believes that someone who answered Agree differs …
Examining The Type I Error And Power Of 18 Common Post-Hoc Comparison Tests, Derek Sauder
Examining The Type I Error And Power Of 18 Common Post-Hoc Comparison Tests, Derek Sauder
Masters Theses, 2010-2019
Researchers utilizing either experimental or quasi-experimental research often want to compare group means. However, with more than two groups, comparing group means may result in an inflated Type I error rate, the probability of wrongly rejecting a null hypothesis. Researchers often employ analysis of variance (ANOVA) methodology to compare more than two group means. Post-hoc comparison procedures (PCPs) are utilized to indicate which group means differ following a significant ANOVA. SPSS provides 18 options for PCPs. The purpose of this study was to determine which PCP provides the best power while maintaining Type I error control when assumptions of ANOVA …
Retrospective Versus Prospective Measurement Of Examinee Motivation In Low-Stakes Testing Contexts: A Moderated Mediation Model, Aaron J. Myers
Retrospective Versus Prospective Measurement Of Examinee Motivation In Low-Stakes Testing Contexts: A Moderated Mediation Model, Aaron J. Myers
Masters Theses, 2010-2019
Expectancy-value theory applied to examinee motivation suggests examinees’ perceived value of a test indirectly affects test performance via examinee effort. This empirically supported indirect effect, however, is often modeled using importance and effort scores measured after test completion, which does not align with their theoretically specified temporal order. Retrospectively measured importance and effort scores may be influenced by examinees’ test performance, impacting the estimate of the indirect effect. To investigate the effect of timing of measurement, first-year college students were randomly assigned to one of three conditions where (1) importance and effort were measured retrospectively; (2) importance was measured prospectively; …
Student Learning Gains In Higher Education: A Longitudinal Analysis With Faculty Discussion, Catherine E. Mathers
Student Learning Gains In Higher Education: A Longitudinal Analysis With Faculty Discussion, Catherine E. Mathers
Masters Theses, 2010-2019
Student learning is the primary desired outcome of a college education. To understand how educational programming and curricula affect students, colleges and universities must collect evidence of student learning gain. In this study, a longitudinal design was employed to investigate how a math and science general education curriculum impacted college students’ quantitative and scientific reasoning. Quantitative and scientific reasoning gain scores were computed and predicted from personal (i.e., prior knowledge, gender) and curriculum (i.e., number of completed courses in the domain) characteristics to uncover what factors relate to learning gain. Collapsing across personal and curriculum variables, gain scores were moderate …
Using Multiple Imputation To Mitigate The Effects Of Low Examinee Motivation On Estimates Of Student Learning, Kelly J. Foelber
Using Multiple Imputation To Mitigate The Effects Of Low Examinee Motivation On Estimates Of Student Learning, Kelly J. Foelber
Dissertations, 2014-2019
In higher education, we often collect data in order to make inferences about student learning, and ultimately, in order to make evidence-based changes to try to improve student learning. The validity of the inferences we make, however, depends on the quality of the data we collect. Low examinee motivation compromises these inferences; research suggests that low examinee motivation can lead to inaccurate estimates of examinees’ ability (e.g., Wise & DeMars, 2005). To obtain data that better represent what students know, think, and can do, practitioners must consider, and attempt to negate the effects of, low examinee motivation. The primary purpose …
You Only Live Up To The Standards You Set: An Evaluation Of Different Approaches To Standard Setting, Scott N. Strickman
You Only Live Up To The Standards You Set: An Evaluation Of Different Approaches To Standard Setting, Scott N. Strickman
Dissertations, 2014-2019
Interpretation of performance in reference to a standard can provide nuanced, finely-tuned information regarding examinee abilities beyond that of just a total score. However, there is a multitude of ways to set performance standards yet little guidance regarding which method operates best and under what circumstances. Traditional methods are the most common approach adopted in practice and heavily involve subject matter experts (SMEs). Two other approaches have been suggested in the literature as alternative ways to set performance standards, although they have yet to be implemented in practice. Data-driven approaches do not involve SMEs but rather rely solely upon statistical …
Examining Latent Change Classes: An Application Of Factor Mixture Modeling To Change Scores, Thai Q. Ong
Examining Latent Change Classes: An Application Of Factor Mixture Modeling To Change Scores, Thai Q. Ong
Masters Theses, 2010-2019
Although change scores are used in a variety of statistical methods (e.g., analysis of variance and regression), there is a lack of application of latent variable modeling methods to change scores. This thesis provides a detailed description of two latent variable modeling methods applied to change scores: factor analysis of change scores and change score factor mixture modeling. To illustrate advantages of these methods, both were applied to change score data from undergraduates. Students responded to sense of identity items during a university-wide assessment day on two occasions, once as incoming freshmen and again as second-semester sophomores. Change scores were …
The Effect Of Anchoring Vignettes On Factor Structures: Student Effort As An Example, Carolyn A. Miesen
The Effect Of Anchoring Vignettes On Factor Structures: Student Effort As An Example, Carolyn A. Miesen
Masters Theses, 2010-2019
Anchoring vignettes are used as a methodological technique for removing differential interpretation of response categories (DIRC) from scores on subjective self-report measures (King, Murray, Slomon, & Tandon, 2004). This technique requires participants to read one or more short scenarios, or vignettes, designed to represent various levels of a construct. Vignette ratings are used as an indication of DIRC, which is a source of differential item functioning (DIF). Prior research primarily used indirect methods for evaluating vignette quality. In response, the present set of studies proposes using invariance testing as a more direct evaluation of how the use of anchoring vignettes …
Applying Solution Behavior Thresholds To A Noncognitive Measure To Identify Rapid Responders: An Empirical Investigation, Mary M. Johnston
Applying Solution Behavior Thresholds To A Noncognitive Measure To Identify Rapid Responders: An Empirical Investigation, Mary M. Johnston
Dissertations, 2014-2019
Noncognitive measures are increasingly being used for accountability purposes in higher education (e.g., O. L. Liu, Frankel, & Roohr, 2014). Because these measures are often collected under low-stakes conditions, there is a concern students do not put forth their best effort when responding, which is problematic given previous research has found noneffortful responding can negatively impact the validity of results (e.g., Barry & Finney, 2009; Meade & Craig, 2012; Swerdzewski, Harmes, & Finney, 2011). Subsequently, there is a need to identify students displaying low effort on low-stakes noncognitive measures. One method, which is based on response time and can discreetly …
Birds Of A Feather Learn Together: Learning Community Outcomes Assessment Using Propensity Score Matching, Elisabeth M. Pyburn, Heather Dawn Harris
Birds Of A Feather Learn Together: Learning Community Outcomes Assessment Using Propensity Score Matching, Elisabeth M. Pyburn, Heather Dawn Harris
Showcase of Graduate Student Scholarship and Creative Activities
The goal of the study was to provide an example of propensity score matching techniques within the context of higher education assessment. At our institution, a select number of incoming first-year students participate in major-specific learning communities. Because the decision to join the communities is purely voluntary, one might expect that students who elect to join the program may differ from those who do not. Thus, important covariates related to self-selection into the learning community were identified. Two years of Arts learning community data were analyzed to compare the academic performance and civic-mindedness of learning community students to an arts …
The Effects Of Ordinal Data On Coefficient Alpha, Kathryn E. Pinder
The Effects Of Ordinal Data On Coefficient Alpha, Kathryn E. Pinder
Masters Theses, 2010-2019
Given coefficient alpha’s wide prevalence as a measure of internal reliability, it is important to know the conditions under which it is an appropriate estimate of reliability. The present paper explores alpha’s assumption of uncorrelated errors when used with ordinal data. Alpha overestimates true reliability when correlated errors are present. In this paper, I use a simulation study to recreate three mechanisms proposed to create correlated errors in ordinal data. The first mechanism, misclassification error, occurs when there are correlated measurement errors present in the data. The second mechanism, grouping error, occurs when there are not enough categories to represent …
Propensity Score Matching In Higher Education Assessment, Heather D. Harris
Propensity Score Matching In Higher Education Assessment, Heather D. Harris
Masters Theses, 2010-2019
The applied nature of higher education assessment does not lend itself to rigorous experimental research designs. However, assessment practitioners would like to make claims about the influence of educational programs on student learning outcomes. Propensity score matching (PSM) methods are quasi-experimental techniques that allow researchers to control for known confounding variables. In the context of higher education, PSM techniques allow assessment practitioners to control for confounding variables related to students’ self-selected participation in university programs. Research and recommendations on how to apply PSM techniques are scattered throughout several disciplines. However, additional research is needed to evaluate how well PSM techniques …
Persons Can Speak Louder Than Variables: Person-Centered Analyses And The Prediction Of Student Success, Elisabeth M. Pyburn
Persons Can Speak Louder Than Variables: Person-Centered Analyses And The Prediction Of Student Success, Elisabeth M. Pyburn
Masters Theses, 2010-2019
In order to ensure that analyses are appropriate for one’s research question(s), it is important to consider whether a person-centered or variable-centered approach is needed. Person-centered approaches are often not considered in situations for which they would be appropriate. To that end, a description of the characteristics and procedures of two common person-centered analyses (cluster analysis and mixture modeling) are provided. Although both analyses accomplish the same general aim – to group persons based on their similarity on a series of variables, thus providing ease of interpretation – the methods employed for each analysis differ considerably. As illustration, both analyses …
The Effects Of A Planned Missingness Design On Examinee Motivation And Psychometric Quality, Matthew S. Swain
The Effects Of A Planned Missingness Design On Examinee Motivation And Psychometric Quality, Matthew S. Swain
Dissertations, 2014-2019
Assessment practitioners in higher education face increasing demands to collect assessment and accountability data to make important inferences about student learning and institutional quality. The validity of these high-stakes decisions is jeopardized, particularly in low-stakes testing contexts, when examinees do not expend sufficient motivation to perform well on the test. This study introduced planned missingness as a potential solution. In planned missingness designs, data on all items are collected but each examinee only completes a subset of items, thus increasing data collection efficiency, reducing examinee burden, and potentially increasing data quality. The current scientific reasoning test served as the Long …
Extending An Irt Mixture Model To Detect Random Responders On Non-Cognitive Polytomously Scored Assessments, Mandalyn R. Swanson
Extending An Irt Mixture Model To Detect Random Responders On Non-Cognitive Polytomously Scored Assessments, Mandalyn R. Swanson
Dissertations, 2014-2019
This study represents an attempt to distinguish two classes of examinees – random responders and valid responders – on non-cognitive assessments in low-stakes testing. The majority of existing literature regarding the detection of random responders in low-stakes settings exists in regard to cognitive tests that are dichotomously scored. However, evidence suggests that random responding occurs on non-cognitive assessments, and as with cognitive measures, the data derived from such measures are used to inform practice. Thus, a threat to test score validity exists if examinees’ response selections do not accurately reflect their underlying level on the construct being assessed. As with …
Examining The Performance Of The Metropolis-Hastings Robbins-Monro Algorithm In The Estimation Of Multilevel Multidimensional Irt Models, Bozhidar M. Bashkov
Examining The Performance Of The Metropolis-Hastings Robbins-Monro Algorithm In The Estimation Of Multilevel Multidimensional Irt Models, Bozhidar M. Bashkov
Dissertations, 2014-2019
The purpose of this study was to review the challenges that exist in the estimation of complex (multidimensional) models applied to complex (multilevel) data and to examine the performance of the recently developed Metropolis-Hastings Robbins-Monro (MH-RM) algorithm (Cai, 2010a, 2010b), designed to overcome these challenges and implemented in both commercial and open-source software programs. Unlike other methods, which either rely on high-dimensional numerical integration or approximation of the entire multidimensional response surface, MH-RM makes use of Fisher’s Identity to employ stochastic imputation (i.e., data augmentation) via the Metropolis-Hastings sampler and then apply the stochastic approximation method of Robbins and Monro …
Addressing Serial-Order And Negative-Keying Effects: A Mixed-Methods Study, Jerusha J. Gerstner
Addressing Serial-Order And Negative-Keying Effects: A Mixed-Methods Study, Jerusha J. Gerstner
Dissertations, 2014-2019
Researchers have studied item serial-order effects on attitudinal instruments by considering how item-total correlations differ based on the item’s placement within a scale (e.g., Hamilton & Shuminsky, 1990). In addition, other researchers have focused on item negative-keying effects on attitudinal instruments (e.g., Marsh, 1996). Researchers consistently have found that negatively-keyed items relate to one another above and beyond their relationship to the construct intended to be measured. However, only one study (i.e., Bandalos & Coleman, 2012) investigated the combined effects of serial-order and negative-keying on attitudinal instruments. Their brief study found some improvements in fit when attitudinal items were presented …