Open Access. Powered by Scholars. Published by Universities.®
- Discipline
- Keyword
-
- Assessment (2)
- Higher education (2)
- Measurement invariance (2)
- AAVE (1)
- AI Literacy (1)
-
- Anxiety (1)
- Bayesian (1)
- COVID-19 (1)
- Church (1)
- Confirmatory factor analysis (1)
- Diagnostic classification modeling (1)
- Distractor analysis (1)
- Distractors (1)
- Expectancy violations (1)
- Family communication (1)
- First-generation college students (1)
- Funnel items (1)
- Generalized boosted modeling (1)
- Generative AI (1)
- Halo (1)
- Higher education assessment (1)
- IRT (1)
- Identity (1)
- Infit (1)
- Leniency (1)
- Low-stakes testing (1)
- Low-stakes tests (1)
- MCMC (1)
- Model misspecification (1)
- Motivation (1)
Articles 1 - 14 of 14
Full-Text Articles in Quantitative Psychology
Examining The Glat As A Measure Of Ai Literacy, Hunter R. Castleton
Examining The Glat As A Measure Of Ai Literacy, Hunter R. Castleton
Masters Theses, 2020-current
The current study examined the Generative AI Literacy Assessment Test (GLAT) as a promising test of AI literacy due to its content, which is grounded in relevant AI literacy theory, and its design as a cognitive measure of AI literacy. To examine the GLAT, data were collected from a sample of 543 graduating seniors at James Madison University (JMU) who took the GLAT as part of a longer series of tests. Their responses were first analyzed to assess the test’s item characteristics and internal structure. Following this initial analysis, a final item set was established to ensure that low-quality items …
Assessing Examinee Effort: The Role Of Test Length In Low-Stakes Testing, Jonathan M. Henriques
Assessing Examinee Effort: The Role Of Test Length In Low-Stakes Testing, Jonathan M. Henriques
Masters Theses, 2020-current
In the context of low stakes testing, where students have no benefits or consequences for their performance, test-taking effort becomes a central validity concern. Recent research has explored how shortening tests might improve effort. With modern planned missingness designs, it is possible to reduce the length of the test for the student without compromising the validity and reliability of the test. In this study, I investigated the impact of test length on effort using rapid guessing, a more contemporary technique that has not been applied to this research area. I randomly assigned students to one of three conditions: the full …
Measuring Alumni Career Outcomes: A Validity Study, Mason Jones
Measuring Alumni Career Outcomes: A Validity Study, Mason Jones
Masters Theses, 2020-current
Higher education experiences routine scrutiny as the cost continues to rise and evidence of a return on investment is variable. Higher education institutions would benefit from measuring post-graduate outcomes to better articulate their impact. The current study examines the factor structure and measurement invariance of the National Alumni Career Mobility (NACM) a 25-item survey, that is widely used to assess alumni career outcomes. Prior psychometric reports for the NACM purported two different factor structures: a five-factor and bifactor model. National reports disaggregated the data by demographic variables, including first-generation status. Disaggregation by first-generation status is problematic given the lack of …
Examining The Psychometric Properties Of Issaq-Ss Subscales And Testing Measurement Invariance Across First-Generation Status, Riley Herr
Masters Theses, 2020-current
The ISSAQ-SS is a new scale developed for the purpose of measuring students on twelve noncognitive skills, all of which are theoretically predictive of academic success in higher education. Each of the twelve noncognitive skills measured via the ISSAQ-SS are represented by distinct subscales which contain five to nine items. Students respond to each item using a 4-point Likert scale. Given that the ISSAQ-SS is still new, further validity evidence is needed to support the use of this scale. For example, prior to comparing ISSAQ-SS subscale scores across groups of interest, it is necessary to establish that the subscales exhibit …
Using Irtrees To Account For Response Style Effects Between Item Formats, Stephanie Leroy
Using Irtrees To Account For Response Style Effects Between Item Formats, Stephanie Leroy
Masters Theses, 2020-current
Response styles are consistent person-traits that are defined as the tendency to systematically select responses unrelated to the construct being measured (Paulhus, 1991). Response styles introduce construct-irrelevant variance that distorts observed scores on a measure and biases interpretation of the data. The current study looks at midpoint response style (MRS) and extreme response style (ERS). MRS is the tendency to select the midpoint of a rating scale, while ERS is the tendency to select the endpoints of a rating scale. Previous research sought to either account for response style effects or prevemt them – the current study does both. To …
Double Dosing: Investigating The Utility Of Multiple Priming Questions On Test-Taking Motivation, Mara Mcfadden
Double Dosing: Investigating The Utility Of Multiple Priming Questions On Test-Taking Motivation, Mara Mcfadden
Masters Theses, 2020-current
Priming examinees with questions about intended effort prior to testing has been shown to significantly increase examinee expended effort via self-reported effort and response-time effort. However, this question-behavior effect seems to wear off later in a testing session, specifically when a test is given second in the session. I examined whether administering a second “dose” of the question-behavior effect could combat the decrease in examinee effort later in a testing session. To evaluate whether “double dosing” could increase examinee effort later in a testing session, I randomly assigned examinees to one of three question conditions prior to completing two low-stakes …
An Exploration Of Higher Education Assessment And The Role Of Diagnostic Classification Modeling In Student Learning Outcomes, Samantha Harmon
An Exploration Of Higher Education Assessment And The Role Of Diagnostic Classification Modeling In Student Learning Outcomes, Samantha Harmon
Masters Theses, 2020-current
Higher education assessment measures student knowledge across multiple specific skills within the general education curriculum. Results are used to inform programmatic improvement, holding universities accountable for satisfying the requirements of the state and/or accrediting agencies. Current methods for assessing student learning in higher education assess institutional objectives through aggregated information using a single total score. The Diagnostic Classification Method (DCM) has been shown to yield fine-grained information about students individually and in the aggregate, providing individual-level and SLO-specific feedback. Unfortunately, few studies have investigated the role of the DCM within higher education assessment of student learning, indicating the utility of …
Writing While Black: African American Vernacular English (Aave) And Perceived Writing Performance, Jaylin N. Nesbitt
Writing While Black: African American Vernacular English (Aave) And Perceived Writing Performance, Jaylin N. Nesbitt
Masters Theses, 2020-current
In the education system, there have historically been inequities that have severely disadvantaged Black students academically. One area in which these inequities surface is on writing assessments in the form of lower scores. I argue that because the U.S. education system is centered around Standard American English (SAE), it disadvantages those from different linguistic backgrounds, specifically Black students, as they are most likely to be speakers of African American Vernacular English (AAVE). Although there are theoretical justifications for this, past literature has not empirically tied inequities on writing assessments to Black students’ use of AAVE. The current study used Natural …
Rapid Response Behavior Before And During The Pandemic, Katarina E. Schaefer
Rapid Response Behavior Before And During The Pandemic, Katarina E. Schaefer
Masters Theses, 2020-current
Different levels of examinee motivation pose a validity threat to the interpretation of test scores. This problem is heightened in low-stakes, remote testing environments. Though some ways exist to gauge average motivation throughout testing, less ways exist to gauge motivation fluctuations throughout a single test. One of those ways is through response times. Specifically, rapid response behavior occurs when examinees quickly answer an item without reading or engaging with the item. At James Madison University (JMU), students participating in campus-wide Assessment Days typically experienced an in-person, proctored Assessment Day. However, that changed during the pandemic. During the pandemic, examinees participated …
Investigating The Self In Self-Report, Samantha L. Boddy
Investigating The Self In Self-Report, Samantha L. Boddy
Masters Theses, 2020-current
Self-report items are ubiquitous in social sciences and services and medical centers. However, there is some concern about whether people are able to accurately report about themselves. One well-known source of concern is social desirability bias (SDB) or socially desirable responding (SDR), which involves people providing overly-positive responses about themselves that better align with social norms than might their actual attitudes or behaviors. However, several researchers (e.g., Brenner & DeLamater, 2016; Hadaway et al., 1998) suggest that a person’s identity in the area of interest may bias their responding. Specifically, that people interpret and respond to items in terms of …
Understanding Motivations To Attend Various Sized Churches: A Study Using Family Communication Patterns, Expectancy Violations, And Anxiety To Predict Church Attendance, Molly Bradshaw
Masters Theses, 2020-current
Two separate studies were conducted to examine whether communication variables impact religious views and church attendance. For the first study, 228 students from a large Southeastern university completed a web survey. The second study was a web survey of 204 adults that was conducted via Amazon Mechanical Turk (MTURK). Both surveys were sent out to determine one’s motivations to attend a small, medium, or large church using family communication, anxiety, expectations, and religion variables as predictors. Family communication, anxiety, and expectancy variables were positively correlated to many aspects of religious views. Hierarchical regression models utilizing demographics, family communication, anxiety, expectancy …
Identifying Rater Effects For Writing And Critical Thinking: Applying The Many-Facets Rasch Model To The Value Institute, Yelisey A. Shapovalov
Identifying Rater Effects For Writing And Critical Thinking: Applying The Many-Facets Rasch Model To The Value Institute, Yelisey A. Shapovalov
Masters Theses, 2020-current
Performance assessments require examinees to carry out a process or produce a product and can be designed to have high fidelity to real-world application of higher-order skills. As such, performance assessments are highly valued in higher education settings. However, performance assessment is vulnerable to psychometric challenges that threaten the validity of scores due to the subjective nature of the scoring process. Specifically, raters must exercise judgement to provide scores to examinee work, which may be impacted by rater effects, or systematic differences in how raters evaluate performance assessment artifacts. Research has indicated that performance assessment may never be fully free …
The Effects Of Undesirable Distractors On Estimates Of Ability, Kathryn N. Thompson
The Effects Of Undesirable Distractors On Estimates Of Ability, Kathryn N. Thompson
Masters Theses, 2020-current
Distractors, or the incorrect options, are an important part of the multiple-choice item. Previous literature has supported the inclusion of distractors when estimating abilities. While the effects of well-functioning distractors on estimates of ability have been examined, research has neglected to examine the effects of undesirable distractors on estimates of ability. Undesirable distractors are defined as distractors that are opposite of what test-developers expect or want distractors to behave. For instance, an upper lure distractor is one that high ability examinees select rather than selecting the correct answer. A simulation study was employed to determine these effects by varying undesirable …
Propensity Score Matching And Generalized Boosted Modeling In The Context Of Model Misspecification: A Simulation Study, Briana G. Craig
Propensity Score Matching And Generalized Boosted Modeling In The Context Of Model Misspecification: A Simulation Study, Briana G. Craig
Masters Theses, 2020-current
In the absence of random assignment, researchers must consider the impact of selection bias – pre-existing covariate differences between groups due to differences among those entering into treatment and those otherwise unable to participate. Propensity score matching (PSM) and generalized boosted modeling (GBM) are two quasi-experimental pre-processing methods that strive to reduce the impact of selection bias before analyzing a treatment effect. PSM and GBM both examine a treatment and comparison group and either match or weight members of those groups to create new, balanced groups. The new, balanced groups theoretically can then be used as a proxy for the …