Quantitative Psychology Commons^™

A Psychometric Analysis Of Natural Language Inference Using Transformer Language Models, Antonio Laverghetta Jr.

USF Tampa Graduate Theses and Dissertations

Large language models (LLMs) are poised to transform both academia and industry. But the excitement around these generative AIs has also been met with concern for the true extent of their capabilities. This dissertation helps to address these questions by examining the capabilities of LLMs using the tools of psychometrics. We focus on analyzing the capabilities of LLMs on the task of natural language inference (NLI), a foundational benchmark often used to evaluate new models. We demonstrate that LLMs can reliably predict the psychometric properties of NLI items were those items administered to humans. Through a series of experiments, we …

A New Method To Determine The Posterior Distribution Of Coefficient Alpha, John Mart V. Delosreyes

Psychology Theses & Dissertations

There is a focus within the behavioral/social sciences on non-physical, psychological constructs (i.e., constructs). These constructs are indirectly measured using measurement instruments that consist of questions that capture the manifestations of these constructs. The indirect nature of measuring constructs results in a need of ensuring that measurement instruments are reliable. The most popular statistic used to estimate reliability is coefficient alpha as it is easy to compute and has properties that make it desirable to use. Coefficient alpha’s popularity has resulted in a wide breadth of research into its qualities. Notably, research about coefficient alpha’s distribution has led to developments …

Asking The Right Questions: Insights Into Assessing Intercultural Sensitivity, Anjana Balakrishnan

Electronic Thesis and Dissertation Repository

Intercultural sensitivity represents a well-studied interdisciplinary construct which is measured using multiple tools. However, more effective measurement methods are possible and also needed. This study was intended to refine a well-known tool, i.e., the Intercultural Sensitivity Scale-ISS. New items were written and tested with existing items. 269 undergraduate students completed questionnaires assessing Big Five personality variables, emotional intelligence, Honesty-Humility, intercultural sensitivity, social desirability, and social dominance orientation. Exploratory factor analyses suggested two plausible final scales: 30-items with four-factors (RISS-V1) and 25-items with three-factors (RISS-V2). Both RISS versions demonstrated full scale, subscale, and test-retest reliability. Social dominance orientation correlated negatively while …

The Reliability And Validity Of The Thin Slice Technique: Observational Research On Video Recorded Medical Interactions, Tanina Suzanne Foster

Wayne State University Dissertations

The Reliability and Validity of the Thin Slice Technique: Observational Research on Video Recorded Medical Interactions

Introduction: Observational research using the thin slice technique has been routinely incorporated in observational research methods, however there is limited evidence supporting use of this technique compared to full interaction coding. The purpose of this study was to determine if this technique could be reliability coded, if ratings are consistent between the first, second and third slice, and if they are indeed representative of full interactions.

Methods: Three 30-second thin slices were sampled from the beginning, middle and end of a full-length video-recorded …

Assessing The Psychometric Properties Of A Self-Efficacy Measure Within A Patient Navigation Research Program, Mariana Arevalo

USF Tampa Graduate Theses and Dissertations

There is a dearth of validated self-efficacy (SE) measures in the field of preventive oncology. The objective of this study is to describe the development and validation of a measure to assess patients' perceived ability to obtain the recommended care following an abnormality suspicious for breast cancer. Guided by a social cognitive theory framework, a 51-item measure was developed to explore perceived capability to obtain follow up care under a number of barriers. A multi-step process was utilized to assess the instrument's psychometric properties. First, cognitive validity assessments with experts were conducted, and these aided in the wording refinement of …

Internal Consistency Of The Self-Perception Profile For Children: Using Covariance Structure Modeling To Overcome The Limitations Of Cronbach's Α, Ian Cero

All Graduate Theses, Dissertations, and Other Capstone Projects

Self-perception is linked to a variety of psychosocial outcomes and its measurement has become a priority across a several disciplines. The Self-Perception Profile for Children (SPP-C) is commonly utilized to measure both global self worth and several important sub-domains of self-perception. Although much research has suggested this instrument possesses good internal consistency, previous investigations have primarily employed Cronbach's α; to estimate the stability of responding across items. This represents an important limitation, as α; is vulnerable to mis-estimation in the presence of correlated errors and non-τ-equivalent indicators, neither of which have been ruled out for the SPP-C. The present investigation …

When Does Fidelity Matter? An Evaluation Of Two Medical Simulation Methods, Nneka Joseph

USF Tampa Graduate Theses and Dissertations

Job or task simulations are used in training when the use of the real task is dangerous or expensive, such as flying aircraft or surgery. This study focused on comparing two types of simulations used in assessments during a Clinical Performance Examination of third-year medical students: computer enhanced mannequins and standardized patients. Each type of simulation has advantages, but little empirical work exists to guide the use of different types of simulation for training and evaluating different aspects of performance. This study analyzed performance scores for different competencies as well as the reliability and validity of the different simulation types. …

Development And Validation Of The Cultural Competence Of Program Evaluators (Ccpe) Scale, Krystall Dunaway

Psychology Theses & Dissertations

As part of its Guiding Principles for Evaluators, the American Evaluation Association (AEA) requires that evaluators develop cultural competencies, yet no measure of cultural competence currently exists in the field. Using items from cultural competence measures used in fields such as counseling and nursing, in conjunction with the creation of qualitative questions, the researcher developed the Cultural Competence of Program Evaluators (CCPE) scale. The main goal of this study was to validate the CCPE, and a subsidiary goal was to assess differences in level of cultural competence among program evaluators based on various demographic variables such as minority status, age, …