Quantitative Psychology Commons^™

A Psychometric Analysis Of Natural Language Inference Using Transformer Language Models, Antonio Laverghetta Jr.

USF Tampa Graduate Theses and Dissertations

Large language models (LLMs) are poised to transform both academia and industry. But the excitement around these generative AIs has also been met with concern for the true extent of their capabilities. This dissertation helps to address these questions by examining the capabilities of LLMs using the tools of psychometrics. We focus on analyzing the capabilities of LLMs on the task of natural language inference (NLI), a foundational benchmark often used to evaluate new models. We demonstrate that LLMs can reliably predict the psychometric properties of NLI items were those items administered to humans. Through a series of experiments, we …

Comparing Measures Of Physical Activity Intensity, Duration, And Frequency Using Receiver Operator Characteristic Curve Analyses, Abigail M. Nehrkorn-Bailey

Graduate Theses, Dissertations, and Problem Reports

The United States Department of Health and Human Services (HHS) recommends adults to engage in weekly moderate- or vigorous-intensity physical activity based on its association with various physical and psychological health benefits (HHS, 2008; Schoenborn, Adams, & Peregoy, 2013). These physical activity recommendations contain important information for three physical activity components: intensity, frequency, and duration. The current physical activity literature contains gaps, with a lack of specificity for which components are being studied. Although some of the literature does describe the physical activity components, there are many discrepancies in the level of agreement across subjective and objective measures, along with …

Beyond Motivation: Differences In Score Meaning Between Assessment Conditions, Nikole Gregg

Masters Theses, 2010-2019

Written communication is a skill necessary for not only the success of undergraduate students, but for post-graduates in the workplace. Furthermore, according to employers the writing skills of post-graduates tend to be below expectations. Therefore, the assessment of such skills within higher education is in high demand. Written communication assessments tend to be administered in one of two conditions: 1) course embedded and 2) a low-stakes, non-embedded condition. The current study investigated possible construct-irrelevant variance in writing assessment scores by using data from a mid-sized public university in the Mid-Atlantic region of the United States. Specifically, 157 student products were …

Mapping Alternative Masculinities: Development, Validation, And Latent Profile Analysis Of A New Masculinity Measure, Jessica K. Padgett

Electronic Thesis and Dissertation Repository

Prominent measures of masculinity focus on traditional masculine norms, such as high aggression, low emotional expression, and heteronormativity. However, recent qualitative research has indicated that a variety of men embrace alternative forms of masculinity that include unique characteristics not represented by traditional norms. I developed the Alternative Masculinity Measure (ALT-M) to address this gap. The ALT-M was designed to measure individual differences on constructs derived from a modern, socially progressive representation of masculinity. Concepts, scales, and items were developed primarily from readings of qualitative research on alternative masculinities. Nine dimensions with 14 items each was sent to 15 experts for …

Communicating Criterion-Related Validity Using Expectancy Charts: A New Approach, Jeffrey M. Cucina, Julia L. Berger, Henry H. Busciglio

Personnel Assessment and Decisions

Often, personnel selection practitioners present the results of their criterion-related validity studies to their senior leaders and other stakeholders when trying to either implement a new test or validate an existing test. It is sometimes challenging to present complex, statistical results to non-statistical audiences in a way that enables intuitive decision making. Therefore, practitioners often turn to expectancy charts to depict criterion-related validity. There are two main approaches for constructing expectancy charts (i.e., use of Taylor-Russell tables or splitting a raw dataset), both of which have considerable limitations. We propose a new approach for creating expectancy charts based on the …

The Construct And Predictive Validity Of Psychosocial Correlates Of Television Viewing, Raheem Paxton, Pascal Jean-Pierre, Saehwan Park, Yong Gao Dr., Stephen Herrmann, G J. Norman

Journal of Health Disparities Research and Practice

Background: Many studies have examined the consequences of prolonged television viewing, but few studies have examined the psychological states that contribute to this behavior. In this study, we evaluated the construct and predictive validity of psychosocial correlates of television viewing in a population of African American (AA) breast cancer survivors (BCS).

Methods: AA BCS (N = 342, Mean age = 54 years) completed measures of decisional balance, self-efficacy, family support, and time spent watching television online. Exploratory structural equation modeling (ESEM) was used to examine the construct and predictive validity as well as the differential item functioning of the instruments …

Asking The Right Questions: Insights Into Assessing Intercultural Sensitivity, Anjana Balakrishnan

Electronic Thesis and Dissertation Repository

Intercultural sensitivity represents a well-studied interdisciplinary construct which is measured using multiple tools. However, more effective measurement methods are possible and also needed. This study was intended to refine a well-known tool, i.e., the Intercultural Sensitivity Scale-ISS. New items were written and tested with existing items. 269 undergraduate students completed questionnaires assessing Big Five personality variables, emotional intelligence, Honesty-Humility, intercultural sensitivity, social desirability, and social dominance orientation. Exploratory factor analyses suggested two plausible final scales: 30-items with four-factors (RISS-V1) and 25-items with three-factors (RISS-V2). Both RISS versions demonstrated full scale, subscale, and test-retest reliability. Social dominance orientation correlated negatively while …

The Nature Of Conflict In Sport: Development And Validation Of The Group Conflict Questionnaire, Kyle F. Paradis

Electronic Thesis and Dissertation Repository

The purpose of the present dissertation was to develop a questionnaire to assess intra-group conflict in sport teams. To this end, the current dissertation consisted of three phases which followed a logical progression that is typical in the questionnaire development process. A total of (N = 752) participants took part in the three phases (Phase 1: N = 10; Phase 2: N = 437; Phase 3: N = 305).

Phase 1 was a qualitative investigation of athletes’ (N = 10) perceptions of the nature of conflict in sport. This phase was undertaken to gain a better understanding of the conflict …

The Reliability And Validity Of The Thin Slice Technique: Observational Research On Video Recorded Medical Interactions, Tanina Suzanne Foster

Wayne State University Dissertations

The Reliability and Validity of the Thin Slice Technique: Observational Research on Video Recorded Medical Interactions

Introduction: Observational research using the thin slice technique has been routinely incorporated in observational research methods, however there is limited evidence supporting use of this technique compared to full interaction coding. The purpose of this study was to determine if this technique could be reliability coded, if ratings are consistent between the first, second and third slice, and if they are indeed representative of full interactions.

Methods: Three 30-second thin slices were sampled from the beginning, middle and end of a full-length video-recorded …

Convergent And Incremental Predictive Validity Of Clinician, Self-Report, And Structured Interview Diagnoses For Personality Disorders Over 5 Years, Douglas B. Samuel, Charles A. Sanislow, Christopher J. Hopwood, M. Tracie Shea, Andrew E. Skodol, Leslie C. Morey, Emily B. Ansell, John C. Markowitz, Mary C. Zanarini, Carlos M. Grilo

Charles A. Sanislow, Ph.D.

OBJECTIVE: Research has demonstrated poor agreement between clinician-assigned personality disorder (PD) diagnoses and those generated by self-report questionnaires and semistructured diagnostic interviews. No research has compared prospectively the predictive validity of these methods. We investigated the convergence of these 3 diagnostic methods and tested their relative and incremental validity in predicting independent, multimethod assessments of psychosocial functioning performed prospectively over 5 years.

METHOD: Participants were 320 patients in the Collaborative Longitudinal Personality Disorders Study diagnosed with PDs by therapist, self-report, and semistructured interview at baseline. We examined the relative incremental validity of therapists' naturalistic ratings relative to these other diagnostic …

Assessing The Psychometric Properties Of A Self-Efficacy Measure Within A Patient Navigation Research Program, Mariana Arevalo

USF Tampa Graduate Theses and Dissertations

There is a dearth of validated self-efficacy (SE) measures in the field of preventive oncology. The objective of this study is to describe the development and validation of a measure to assess patients' perceived ability to obtain the recommended care following an abnormality suspicious for breast cancer. Guided by a social cognitive theory framework, a 51-item measure was developed to explore perceived capability to obtain follow up care under a number of barriers. A multi-step process was utilized to assess the instrument's psychometric properties. First, cognitive validity assessments with experts were conducted, and these aided in the wording refinement of …

An Investigation Of Measurement Invariance Across Genders On The Overexcitability Questionnaire-Two (Oeqii), Russell Warne

Russell T Warne

The Overexcitability Questionnaire–Two (OEQII) is a quantitative instrument for assessing overexcitabilities as they are described in Dabrowski’s theory of positive disintegration. This article uses multigroup confirmatory factor analysis to examine the measurement invariance of OEQII scores across genders. Results indicate that raw OEQII scores cannot be compared across genders. Caution should be used in interpreting OEQII scores.

When Does Fidelity Matter? An Evaluation Of Two Medical Simulation Methods, Nneka Joseph

USF Tampa Graduate Theses and Dissertations

Job or task simulations are used in training when the use of the real task is dangerous or expensive, such as flying aircraft or surgery. This study focused on comparing two types of simulations used in assessments during a Clinical Performance Examination of third-year medical students: computer enhanced mannequins and standardized patients. Each type of simulation has advantages, but little empirical work exists to guide the use of different types of simulation for training and evaluating different aspects of performance. This study analyzed performance scores for different competencies as well as the reliability and validity of the different simulation types. …

Massachusetts Youth Screening Instrument Long-Term Outcomes And Scale Stability, Elise Christina Simonds Bisbee

Psychology Theses & Dissertations

The Massachusetts Youth Screening Instrument-2 (MAYSI-2; Grisso & Barnum, 2006) was developed in 1998 to offer an efficient measure for identifying adolescents within the juvenile justice system in need of further psychiatric evaluation, treatment, or specialized care. Since the instrument's publication, several studies have evaluated the psychometric properties and clinical utility of the MAYSI-2. The current study adds to the literature examining the reliability and validity of this measure. Specifically, the current study sought to evaluate the long-term characteristics and predictive utility of the MAYSI-2 scale scores. This study utilized a sample of 8,929 boys (n = 6.780) and …

Development And Validation Of The Cultural Competence Of Program Evaluators (Ccpe) Scale, Krystall Dunaway

Psychology Theses & Dissertations

As part of its Guiding Principles for Evaluators, the American Evaluation Association (AEA) requires that evaluators develop cultural competencies, yet no measure of cultural competence currently exists in the field. Using items from cultural competence measures used in fields such as counseling and nursing, in conjunction with the creation of qualitative questions, the researcher developed the Cultural Competence of Program Evaluators (CCPE) scale. The main goal of this study was to validate the CCPE, and a subsidiary goal was to assess differences in level of cultural competence among program evaluators based on various demographic variables such as minority status, age, …