Quantitative Psychology Commons^™

Examining The Glat As A Measure Of Ai Literacy, Hunter R. Castleton

Masters Theses, 2020-current

The current study examined the Generative AI Literacy Assessment Test (GLAT) as a promising test of AI literacy due to its content, which is grounded in relevant AI literacy theory, and its design as a cognitive measure of AI literacy. To examine the GLAT, data were collected from a sample of 543 graduating seniors at James Madison University (JMU) who took the GLAT as part of a longer series of tests. Their responses were first analyzed to assess the test’s item characteristics and internal structure. Following this initial analysis, a final item set was established to ensure that low-quality items …

The Impact Of Free Vs. Fixed Factor Correlations On Parameter Recovery In Multidimensional Irt Models, Stephanie Leroy

Dissertations, 2020-current

This dissertation investigated how cross-loading specification and factor-correlation treatment influence parameter recovery in multidimensional item response theory (MIRT) models. Two simulation studies extended Hosseinzadeh and Cole (2024) by examining whether freely estimating factor correlations improves recovery of item parameters, factor correlations, and latent traits. Study 1 used a compensatory multidimensional two-parameter logistic (M2PL) model with two correlated dimensions and 20 dichotomous items. Manipulated conditions included cross-loading specification (correctly specified vs. ignored), proportion of cross-loaded items, magnitude of cross-loadings, true factor correlation, and factor-correlation treatment during estimation (freely estimated vs. fixed to zero). Study 2 extended the design to a multidimensional …

Assessing Examinee Effort: The Role Of Test Length In Low-Stakes Testing, Jonathan M. Henriques

Masters Theses, 2020-current

In the context of low stakes testing, where students have no benefits or consequences for their performance, test-taking effort becomes a central validity concern. Recent research has explored how shortening tests might improve effort. With modern planned missingness designs, it is possible to reduce the length of the test for the student without compromising the validity and reliability of the test. In this study, I investigated the impact of test length on effort using rapid guessing, a more contemporary technique that has not been applied to this research area. I randomly assigned students to one of three conditions: the full …

Investigating The Direction Of Item Parameter Bias In The Presence Of Rapid Guessing, Sarah Alahmadi

Dissertations, 2020-current

Inadequate examinee test-taking effort is especially concerning when the results of low-stakes tests are used to inform high-stakes policy or psychometric decisions. Although abundant is the literature on the negative impact of low test-taking effort (e.g., Wise & DeMars, 2005) and ways to address it (e.g., Alahmadi & DeMars, 2024; Lau et al., 2009), integrating it into assessment practices has long been in a rudimentary stage. The findings presented in this dissertation serve as a clarion call for testing professionals to reinforce efforts that bridge research and practice. The current study investigated how rapid guessing (RG)—a form of non-effortful test-taking …

Measuring Alumni Career Outcomes: A Validity Study, Mason Jones

Masters Theses, 2020-current

Higher education experiences routine scrutiny as the cost continues to rise and evidence of a return on investment is variable. Higher education institutions would benefit from measuring post-graduate outcomes to better articulate their impact. The current study examines the factor structure and measurement invariance of the National Alumni Career Mobility (NACM) a 25-item survey, that is widely used to assess alumni career outcomes. Prior psychometric reports for the NACM purported two different factor structures: a five-factor and bifactor model. National reports disaggregated the data by demographic variables, including first-generation status. Disaggregation by first-generation status is problematic given the lack of …

Examining The Psychometric Properties Of Issaq-Ss Subscales And Testing Measurement Invariance Across First-Generation Status, Riley Herr

Masters Theses, 2020-current

The ISSAQ-SS is a new scale developed for the purpose of measuring students on twelve noncognitive skills, all of which are theoretically predictive of academic success in higher education. Each of the twelve noncognitive skills measured via the ISSAQ-SS are represented by distinct subscales which contain five to nine items. Students respond to each item using a 4-point Likert scale. Given that the ISSAQ-SS is still new, further validity evidence is needed to support the use of this scale. For example, prior to comparing ISSAQ-SS subscale scores across groups of interest, it is necessary to establish that the subscales exhibit …

A Novel Examination Of None-Of-The-Above As It Influences Examinee Item Responses, Kathryn N. Thompson

Dissertations, 2020-current

It is imperative to collect validity evidence prior to interpreting and using test scores. During the process of collecting validity evidence, test developers should consider whether test scores are contaminated by sources of extraneous information. This is referred to as construct irrelevant variance, or the “degree to which test scores are affected by processes that are extraneous to the test’s intended purpose” (AERA et al., 2014, p. 12). One possible source of construct irrelevant variance is violating item-writing guidelines, such as to “avoid the use of none-of-the-above” in multiple-choice items (Rodriguez, 2016, p. 268).

Numerous studies have been conducted with …

Using Irtrees To Account For Response Style Effects Between Item Formats, Stephanie Leroy

Masters Theses, 2020-current

Response styles are consistent person-traits that are defined as the tendency to systematically select responses unrelated to the construct being measured (Paulhus, 1991). Response styles introduce construct-irrelevant variance that distorts observed scores on a measure and biases interpretation of the data. The current study looks at midpoint response style (MRS) and extreme response style (ERS). MRS is the tendency to select the midpoint of a rating scale, while ERS is the tendency to select the endpoints of a rating scale. Previous research sought to either account for response style effects or prevemt them – the current study does both. To …

Double Dosing: Investigating The Utility Of Multiple Priming Questions On Test-Taking Motivation, Mara Mcfadden

Masters Theses, 2020-current

Priming examinees with questions about intended effort prior to testing has been shown to significantly increase examinee expended effort via self-reported effort and response-time effort. However, this question-behavior effect seems to wear off later in a testing session, specifically when a test is given second in the session. I examined whether administering a second “dose” of the question-behavior effect could combat the decrease in examinee effort later in a testing session. To evaluate whether “double dosing” could increase examinee effort later in a testing session, I randomly assigned examinees to one of three question conditions prior to completing two low-stakes …

Many-Facet Rasch Designs: How Should Raters Be Assigned To Examinees?, Christine E. Demars, Yelisey A. Shapovalov, John D. Hathcoat

Department of Graduate Psychology - Faculty Scholarship

In Facets models, raters should be connected, and there are multiple ways to connect raters. Keeping the number of ratings constant and two raters scoring each examinee, the standard error of both rater severity and examinee ability was higher when raters scored one examinee in common with many different raters than when they scored many examinees in common with two raters. However, the differences were small, especially for the standard error of examinee ability. Alternatively, when only a subset of examinees were scored by two or more raters, the smallest standard errors were achieved when all raters scored a common …

An Exploration Of Higher Education Assessment And The Role Of Diagnostic Classification Modeling In Student Learning Outcomes, Samantha Harmon

Masters Theses, 2020-current

Higher education assessment measures student knowledge across multiple specific skills within the general education curriculum. Results are used to inform programmatic improvement, holding universities accountable for satisfying the requirements of the state and/or accrediting agencies. Current methods for assessing student learning in higher education assess institutional objectives through aggregated information using a single total score. The Diagnostic Classification Method (DCM) has been shown to yield fine-grained information about students individually and in the aggregate, providing individual-level and SLO-specific feedback. Unfortunately, few studies have investigated the role of the DCM within higher education assessment of student learning, indicating the utility of …

The Use Of Complex-Structure Items In Multistage Testing, Paulius Satkus

Dissertations, 2020-current

When developing tests, measurement experts may prefer simple-structure items because they measure one trait, which simplifies scoring and scoring interpretation. Conversely, complex-structure items may be preferred to reflect the complexity of multidimensional constructs. The current study sought to address the gap in the literature of multi-stage testing by conducting a simulation study with a hypothetical two-stage adaptive test with a purpose of comparing the performance of simple and complex structure items. The findings suggest that with a longer test (60 items), the two types of items performed similarly with respect to bias and RMSE of the trait estimates. For the …

Writing While Black: African American Vernacular English (Aave) And Perceived Writing Performance, Jaylin N. Nesbitt

Masters Theses, 2020-current

In the education system, there have historically been inequities that have severely disadvantaged Black students academically. One area in which these inequities surface is on writing assessments in the form of lower scores. I argue that because the U.S. education system is centered around Standard American English (SAE), it disadvantages those from different linguistic backgrounds, specifically Black students, as they are most likely to be speakers of African American Vernacular English (AAVE). Although there are theoretical justifications for this, past literature has not empirically tied inequities on writing assessments to Black students’ use of AAVE. The current study used Natural …

Rapid Response Behavior Before And During The Pandemic, Katarina E. Schaefer

Masters Theses, 2020-current

Different levels of examinee motivation pose a validity threat to the interpretation of test scores. This problem is heightened in low-stakes, remote testing environments. Though some ways exist to gauge average motivation throughout testing, less ways exist to gauge motivation fluctuations throughout a single test. One of those ways is through response times. Specifically, rapid response behavior occurs when examinees quickly answer an item without reading or engaging with the item. At James Madison University (JMU), students participating in campus-wide Assessment Days typically experienced an in-person, proctored Assessment Day. However, that changed during the pandemic. During the pandemic, examinees participated …

Differential Motivation In Remote Educational Assessment: Person-Based Filtering Versus Response-Based Filtering, Sarah Alahmadi, Christine E. Demars

Department of Graduate Psychology - Faculty Scholarship

Large-scale educational assessments are often considered low-stakes, increasing the possibility of confounding true performance level with low motivation. These concerns are amplified in remote testing conditions. To remove the effects of low effort levels in responses observed in remote low-stakes testing, several motivation filtering methods can be used to purify the data. We estimated scores from assessment data collected remotely in Spring 2021 six ways, applying examinee-based filtering methods (filtering examinees based on total time) and response-based filtering methods (filtering responses using the effort-moderated IRT model), varying the thresholds selected to separate solution behavior (SB) responses from rapid-guessing behavior (RGB). …

Item Parameter Recovery With And Without The Use Of Priors, Paulius Satkus, Christine E. Demars

Department of Graduate Psychology - Faculty Scholarship

Marginal maximum likelihood (MML), a common estimation method for IRT models, is not inherently a Bayesian procedure. However, due to estimation difficulties, Bayesian priors are often applied to the likelihood when estimating 3PL models, especially with small samples. Little focus has been placed on choosing the priors for MML estimation. In this study, using samples sizes of 1000 or smaller, not using priors often led to extreme, implausible parameter estimates. Applying prior distributions to the c-parameters alleviated the estimation problems with samples of 1000; priors on both the a-parameters and c-parameters were needed for the samples of …

Investigating The Self In Self-Report, Samantha L. Boddy

Masters Theses, 2020-current

Self-report items are ubiquitous in social sciences and services and medical centers. However, there is some concern about whether people are able to accurately report about themselves. One well-known source of concern is social desirability bias (SDB) or socially desirable responding (SDR), which involves people providing overly-positive responses about themselves that better align with social norms than might their actual attitudes or behaviors. However, several researchers (e.g., Brenner & DeLamater, 2016; Hadaway et al., 1998) suggest that a person’s identity in the area of interest may bias their responding. Specifically, that people interpret and respond to items in terms of …

Does Coding Method Matter? An Examination Of Propensity Score Methods When The Treatment Group Is Larger Than The Comparison Group, Beth A. Perkins

Dissertations, 2020-current

In educational contexts, students often self-select into specific interventions (e.g., courses, majors, extracurricular programming). When students self-select into an intervention, systematic group differences may impact the validity of inferences made regarding the effect of the intervention. Propensity score methods are commonly used to reduce selection bias in estimates of treatment effects. In educational contexts, often a larger number of students receive a treatment than not. However, recommendations regarding the application of propensity score methods when the treatment group is larger than the comparison group have not been empirically examined. The current study examined the recommendation to recode the treatment and …

Getting Caught-Up In The Process: Does It Really Matter?, Nikole Gregg

Dissertations, 2020-current

Likert items are the most commonly used item-type for measuring attitudes and beliefs. However, responses from Likert items are often plagued with construct-irrelevant variance due to response style behavior. In other words, variability from Likert-item scores can be parsed into: 1) variance pertinent to the construct or trait of interest, and 2) variance irrelevant to the construct or trait of interest. Multidimensional Item Response Theory (MIRT) is an increasingly common modeling approach to parse out information regarding the response style traits and the trait of interest. These MIRT approaches are categorized into threshold-based approaches and response process approaches. An increasingly …

Understanding Motivations To Attend Various Sized Churches: A Study Using Family Communication Patterns, Expectancy Violations, And Anxiety To Predict Church Attendance, Molly Bradshaw

Masters Theses, 2020-current

Two separate studies were conducted to examine whether communication variables impact religious views and church attendance. For the first study, 228 students from a large Southeastern university completed a web survey. The second study was a web survey of 204 adults that was conducted via Amazon Mechanical Turk (MTURK). Both surveys were sent out to determine one’s motivations to attend a small, medium, or large church using family communication, anxiety, expectations, and religion variables as predictors. Family communication, anxiety, and expectancy variables were positively correlated to many aspects of religious views. Hierarchical regression models utilizing demographics, family communication, anxiety, expectancy …

Identifying Rater Effects For Writing And Critical Thinking: Applying The Many-Facets Rasch Model To The Value Institute, Yelisey A. Shapovalov

Masters Theses, 2020-current

Performance assessments require examinees to carry out a process or produce a product and can be designed to have high fidelity to real-world application of higher-order skills. As such, performance assessments are highly valued in higher education settings. However, performance assessment is vulnerable to psychometric challenges that threaten the validity of scores due to the subjective nature of the scoring process. Specifically, raters must exercise judgement to provide scores to examinee work, which may be impacted by rater effects, or systematic differences in how raters evaluate performance assessment artifacts. Research has indicated that performance assessment may never be fully free …

The Effects Of Undesirable Distractors On Estimates Of Ability, Kathryn N. Thompson

Masters Theses, 2020-current

Distractors, or the incorrect options, are an important part of the multiple-choice item. Previous literature has supported the inclusion of distractors when estimating abilities. While the effects of well-functioning distractors on estimates of ability have been examined, research has neglected to examine the effects of undesirable distractors on estimates of ability. Undesirable distractors are defined as distractors that are opposite of what test-developers expect or want distractors to behave. For instance, an upper lure distractor is one that high ability examinees select rather than selecting the correct answer. A simulation study was employed to determine these effects by varying undesirable …

Propensity Score Matching And Generalized Boosted Modeling In The Context Of Model Misspecification: A Simulation Study, Briana G. Craig

Masters Theses, 2020-current

In the absence of random assignment, researchers must consider the impact of selection bias – pre-existing covariate differences between groups due to differences among those entering into treatment and those otherwise unable to participate. Propensity score matching (PSM) and generalized boosted modeling (GBM) are two quasi-experimental pre-processing methods that strive to reduce the impact of selection bias before analyzing a treatment effect. PSM and GBM both examine a treatment and comparison group and either match or weight members of those groups to create new, balanced groups. The new, balanced groups theoretically can then be used as a proxy for the …

Examining The Performance Of The Alignment Method In Dif Analyses, Paulius Satkus, Christine E. Demars

Department of Graduate Psychology - Faculty Scholarship

The alignment procedure is a new method for multiple group invariance models. An important advantage of alignment over the traditional methods is that alignment does not require full measurement invariance to estimate group means and variances (Muthén & Asparouhov, 2014). Simulation studies have supported that alignment performs adequately in situations when few items are noninvariant (or function differentially across groups – DIF). In most other studies, the tests were simulated to represent attitudinal surveys (e.g., fewer items, continuous data). In this study, we evaluated how alignment would perform with a typical educational cognitive test – 40 items scored dichotomously. Different …

Examining The Effects Of Specifying Bayesian Priors On The Wald's Test For Dif, Paulius Satkus, Christine E. Demars

Department of Graduate Psychology - Faculty Scholarship

No abstract provided.

An Applied Example Of A Two-Tier Multiple-Group Testlet Model, Paulius Satkus, Christine E. Demars

Department of Graduate Psychology - Faculty Scholarship

No abstract provided.

Test Emotions, Value, And Self-Efficacy: A Longitudinal Model Predicting Examinee Effort And Performance On A Low-Stakes Test, Paulius Satkus

Masters Theses, 2010-2019

The validity of scores from low-stakes tests may be compromised by examinee motivation. Expectancy-Value theory (EV) has been used to frame the antecedents of examinee motivation in low-stakes testing contexts. According to EV theory, the perceived value of the test and the expectancy to succeed on the test directly affect examinee effort, which then affects test performance. Cross-sectional research studies in low-stakes testing contexts offer some support of EV theory. Control-Value theory (CV) serves as another theory to understand motivation toward a task. CV theory encompasses the constructs of expectancy and value from EV theory, but incorporates test emotions as …

Are All Cognitive Items Equally Prone To Position Effects? Exploring The Relationships Among Item Features And Position Effects, Thai Quang Ong

Dissertations, 2014-2019

One type of context effect is a position effect, which implies parameters of an item are influenced by the position of the item on the test. Researchers often discuss two types of position effects: negative position effects and positive position effects (e.g., Albano, 2013; Debeer & Janssen, 2013). Items exhibiting negative position effects become harder when placed later on the test, whereas items exhibiting positive position effects become easier when placed later on the test. Researchers have primarily examined the underlying causes of position effects through an item or person perspective (e.g., Bulut, 2015; Kingston & Dorans, 1984; Qian, 2014). …

Considerations In S-Χ2: Rest Score Or Summed Score, Priors, And Violations Of Normality, Christine E. Demars, Derek Sauder

Department of Graduate Psychology - Faculty Scholarship

The S-χ2 item fit index is one of the few item fit indices that appears to maintain accurate Type I error rates. This study explored grouping examinees by the rest score or summed score, prior distributions for the item parameters, and the shape of the ability distribution. Type I error was slightly closer to the nominal level for the total-score S-χ2 for the longest tests, but power was higher for the rest-score S-χ2 in every condition where power was < 1. Prior distributions reduced the proportion of estimates with extreme standard errors but slightly inflated the Type I error rates in some conditions. When the ability distribution was not normally distributed, integrating over an empirically-estimated distribution yielded Type I error rates closer to the nominal value than integrating over a normal distribution.

The Psychology Of Performance In Elite Youth Soccer Players, Matthew Best

Senior Honors Projects, 2010-2019

This study is a holistic assessment of psychological mindsets, which are one’s attitudes, beliefs, and perceptions, in elite youth male soccer players between the ages of 13 and 18 and the exploration of the relationships between these mindsets and performance outcomes. The mindsets that were assessed were expectancy, growth mindset, value, goals, belongingness, grit, and self-regulation, and the performance outcomes were minutes played, goals scored, and goals allowed. The mindsets were selected through a review of research in education and sport. I conducted Exploratory Factor Analyses (EFA) and Cronbach’s alpha coefficient analyses to assess the validity and reliability of the …