Psychology Commons^™

A Psychometric Analysis Of Natural Language Inference Using Transformer Language Models, Antonio Laverghetta Jr.

USF Tampa Graduate Theses and Dissertations

Large language models (LLMs) are poised to transform both academia and industry. But the excitement around these generative AIs has also been met with concern for the true extent of their capabilities. This dissertation helps to address these questions by examining the capabilities of LLMs using the tools of psychometrics. We focus on analyzing the capabilities of LLMs on the task of natural language inference (NLI), a foundational benchmark often used to evaluate new models. We demonstrate that LLMs can reliably predict the psychometric properties of NLI items were those items administered to humans. Through a series of experiments, we …

Is Chatgpt Good At Establishing Validity?, Travis Clark

AI Assignment Library

Like many researchers, I am still struggling to understand what parts of my job can be done with the help of Large Language Models (LLMs) such as Chat-GPT. I gave Chat-GPT a task that is relatively simple for Research Methods students to do manually. I asked ChatGPT to give me a research study involving the influence of bystanders on prosocial behavior. How did it do?

Psychometric Properties Of The Revised Children’S Anxiety And Depression Scale (Rcads) For Autistic Youth Without Co-Occurring Intellectual Disability, Nasim Khalfe, Amy R. Goetz, Erika S. Trent, Andrew G. Guzick, Orri Smarason, Minjee Kook, Sean Olsen, Ana C. Ramirez, Saira A. Weinzimmer, Leandra Berry

Psychological Science Faculty Publications and Presentations

Autistic youth often present with comorbid anxiety and depression yet there is a dearth of validated assessment tools. The Revised Children’s Anxiety and Depression Scale (RCADS) assesses internalizing symptoms but there is little psychometric data in autistic youth. Treatment-seeking autistic youth with anxiety or obsessive-compulsive symptoms (N = 74; age 6–14 years), and caregivers, were administered the RCADS-Parent, RCADS-Child, and assessments of internalizing, externalizing symptoms and social impairment indicative of autism. RCADS-Parent and RCADS-Child total anxiety scores demonstrated excellent internal consistency, and the six subscales demonstrated acceptable-to-good internal consistency. The RCADS-Child and Parent total anxiety scores were weakly …

Arousing Motives Or Eliciting Stories? On The Role Of Pictures In A Picture–Story Exercise, Philipp Schäpers, Stefan Krumm, Filip Lievens, Nikola Stenzel

Research Collection Lee Kong Chian School Of Business

Picture–story exercises (PSE) form a popular measurement approach that has been widely used for the assessment of implicit motives. However, current theorizing offers two diverging perspectives on the role of pictures in PSEs: either to elicit stories or to arouse motives. In the current study, we tested these perspectives in an experimental design. We administered a PSE either with or without pictures. Results from N = 281 participants revealed that the experimental manipulation had a medium to large effect for the affiliation and power motive domains, but no effect for the achievement motive domain. We conclude that the herein chosen …

Systematic Review Of Reflection Spectroscopy-Based Skin Carotenoid Assessment In Children, Saima Hasnin, Dipti Dev, Taren Swindle, Susan B. Sisson, Stephanie Jilcott Pitts, Tirna Purkait, Shari C. Clifton, Jocelyn Dixon, Virginia C. Stage

Department of Child, Youth, and Family Studies: Faculty Publications

Assessing children’s skin carotenoid score (SCS) using reflection spectroscopy (RS) is a non-invasive, widely used method to approximate fruit and vegetable consumption (FVC). The aims for the current review were to (1) identify distributions of SCS across demographic groups, (2) identify potential non-dietary correlates for RS-based SCS, (3) summarize the validity and reliability of RSbased SCS assessment, and (4) conduct meta-analyses of studies examining the correlation between RS-based SCS with FVC. A literature search in eight databases in June 2021 resulted in 4880 citations and peer-reviewed publications written in English that investigated children’s (2–10 years old) SCS using RS. We …

Utility Of An Emotion Coding System For Parent-Child Interaction Therapy With Toddlers, Christopher Kyle Owen

Graduate Theses, Dissertations, and Problem Reports

Numerous efficacious early interventions target and alter caregiver-child interactions to promote optimal social-emotional outcomes for young children (Bagner et al., 2014). However, research has primarily relied on the use of caregiver report to assess caregiver-child emotion-focused practices, revealing the need for a behavioral observation assessment (Zinsser et al., 2021). Preliminary evidence suggests that Parent-Child Interaction Therapy with Toddlers (PCIT-T) is a well-received and efficacious intervention for reducing disruptive behaviors, improving child internalizing and externalizing behavior, reducing parental stress, and increasing parental sensitivity (Kohlhoff et al., 2021; Kohlhoff, Cibralic, & Morgan, 2020). PCIT-T strives to train caregivers to interact with their …

Revisiting Meta-Analytic Estimates Of Validity In Personnel Selection: Addressing Systematic Overcorrection For Restriction Of Range, Paul R. Sackett, Charlene Zhang, Christopher M. Berry, Filip Lievens

Research Collection Lee Kong Chian School Of Business

This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor–criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic …

Assessing The Validity And Reliability Of The Performance Diagnostic Checklist-Human Services 1.1, Sebastian Jimenez

Theses and Dissertations

The PDC-HS is an informant-based assessment tool used by OBM practitioners in human service settings. An updated version known as the PDC-HS 1.1 has recently been released. The present study measured the validity and reliability of the PDCHS 1.1 by analyzing answers obtained while watching a video of a simulated interview between a consultant and a supervisor. Three video vignettes were created, each describing a performance concern in one or more areas of the PDCHS 1.1. Twenty-one participants watched all vignettes and filled out the PDC-HS 1.1 based on the answers provided. Validity was measured by calculating the percentage of …

Validity Of Neural Distance Measures In Representational Similarity Analysis, Fabian A. Soto, Emily R. Martin, Hyeonjeong Lee, Nafiz Ahmed, Juan Estepa, Kianoosh Hosseini, Olivia A. Stibolt, Valentina Roldan, Alycia Winters, Mohammadreza Bayat

MODVIS Workshop

No abstract provided.

The Reliability And Validity Of The Open Enneagram Of Personality Scales, Kayleigh Kastelein

ELAIA

The purpose of this study is to assess the reliability and validity of the thirty-six-item Open Enneagram of Personality Scales (OEPS). Our general hypothesis was that the OEPS would show adequate reliability evidence but not validity evidence. Participants were acquired through a small denominationally affiliated Midwest university, Amazon Mechanical Turk, and social media. Test-retest reliability was done with 249 participants, whereas internal consistency reliability, factor analysis, and correlations with the Big Five Inventory (BFI; John, Donahue, & Kentle, 1991) were done using 1,039 participants. An average Pearson’s correlation of .68 (range: 0.54 - 0.75) showed inadequate test-retest reliability for the …

The Effects Of Rater Training On Rater Effects And Validity Of Direct Behavior Ratings, Abigail E. Pruitt

Graduate Research Theses & Dissertations

Educators have a responsibility to accurately measure student behavior in order to identify students in need of additional behavioral support. Current behavior screening tools can be lengthy or difficult to complete, and Direct Behavior Ratings (DBRs) offer a solution. However, DBRs are rater-mediated assessments, prone to rater effects. Rater training methods can be used to mitigate these rater effects, but previous research has not investigated the best training method for reduction of rater effects. Additionally, Many Facet Rasch Measurement (MFRM) provides an opportunity to adjust student ratings in response to individual rater tendencies of severity and leniency. Therefore, the primary …

Object Permanence And The Relationship To Sitting Development In Infants With Motor Delays, Mihee An, Emily C. Marcinowski, Lin-Ya Hsu, Jaclynn Stankus, Karl L. Jancart, Michele A. Lobo, Stacy C. Dusing, Sarah W. Mccoy, James A. Bovaird, Sandra Willett, Regina T. Harbourne

Department of Educational Psychology: Faculty Publications

Purpose: This study examines object permanence development in infants with motor delays (MD) compared with infants with typical development (TD) and in relation to sitting skill. Methods: Fifty-six infants with MD (mean age = 10 months) and 36 with TD (mean age = 5.7 months) were assessed at baseline and then at 1.5, 3, and 6 months postbaseline. A scale was developed to measure object permanence (Object Permanence Scale [OPS]), and the Gross Motor Function Measure sitting subsection (GMFM-SS), and the Bayley Scales of Infant and Toddler Development, 3rd Edition (Bayley-III) were administered. Results: Interrater reliability of the OPS was …

Assessing Alexithymia In Forensic Settings: Psychometric Properties Of The 20-Item Toronto Alexithymia Scale Among Incarcerated Adult Offenders, David A. Preece, Cate L. Parry, Maria [Ricks] M. Allan, Alfred Allan

Research outputs 2014 to 2021

Background:

Alexithymia is a trait involving difficulty identifying feelings (DIF), difficulty describing feelings (DDF) and externally orientated thinking (EOT). It is a risk factor for criminal behaviour. It is commonly assessed with the Toronto Alexithymia Scale (TAS-20), but the psychometrics of the TAS-20 have not been tested across the range of offender populations, and it has been suggested it might be unsuitable in incarcerated offenders. Aim: To establish the psychometrics of the TAS-20 among incarcerated offenders.

Methods:

Factorial validity was examined using confirmatory factor analyses, and the invariance of this factor structure was tested against a published community sample. Reliability …

Applicant Faking On Personality Tests: Good Or Bad And Why Should We Care?, Robert P. Tett, Daniel V. Simonet

Personnel Assessment and Decisions

The unitarian understanding of construct validity holds that deliberate response distortion in completing self-report personality tests (i.e., faking) threatens trait-based inferences drawn from test scores. This “faking-is-bad” (FIB) perspective is being challenged by an emerging “faking-is-good” (FIG) position that condones or favors faking and its underlying attributes (e.g., social skill, ATIC) to the degree they contribute to predictor–criterion correlations and are job relevant. Based on the unitarian model of validity and relevant empirical evidence, we argue the FIG perspective is psychometrically flawed and counterproductive to personality-based selection targeting trait-based fit. Carrying forward both positions leads to variously dark futures for …

Faking And The Validity Of Personality Tests: An Experimental Investigation Using Modern Forced Choice Measures, Christopher R. Huber, Nathan R. Kuncel, Katie B. Huber, Anthony S. Boyce

Personnel Assessment and Decisions

Despite the established validity of personality measures for personnel selection, their susceptibility to faking has been a persistent concern. However, the lack of studies that combine generalizability with experimental control makes it difficult to determine the effects of applicant faking. This study addressed this deficit in two ways. First, we compared a subtle incentive to fake with the explicit “fake-good” instructions used in most faking experiments. Second, we compared standard Likert scales to multidimensional forced choice (MFC) scales designed to resist deception, including more and less fakable versions of the same MFC inventory. MFC scales substantially reduced motivated score elevation …

The Reliability And Validity Of The Open Enneagram Of Personality Scales, Kayleigh Kastelein

Honors Program Projects

The purpose of this study is to assess the reliability and validity of the 36-item Open Enneagram of Personality Scales (OEPS). Our general hypothesis was that the OEPS would show adequate reliability evidence but not validity evidence. Participants were acquired through a small denominationally affiliated Midwest university, Amazon Mechanical Turk, and social media. Test-retest reliability was done with 249 participants while internal consistency reliability, factor analysis, and correlations with the Big Five Inventory (BFI; John, Donahue, & Kentle) were done using 1039 participants. An average Pearson’s correlation of .68 (range: 0.54 - 0.75) showed inadequate test-retest reliability for the OEPS …

Criterion Validity Of The Ohio State University Traumatic Brain Injury Identification Method In A Criminal Justice Sample, Sarah M. Monaghan

Student Theses

The purpose of the current study was to determine the clinical utility of the Ohio State University Traumatic Brain Injury Identification Method (OSU-TBI-ID) in a criminal justice sample, by evaluating the criterion-related validity of this instrument. It was hypothesized that this tool could differentiate between incarcerated individuals with or without a history of Traumatic Brain Injury (TBI) on measures evaluating important TBI-related sequalae. The sample consisted of 95 incarcerated men detained at a private correctional facility in a Mid-Atlantic state. Measures used in this study to evaluate executive functioning difficulties, psychiatric difficulties, substance use problems, institutional misconduct and recidivism were …

The Reliability And Validity Of The Open Enneagram Of Personality Scales, Kayleigh Kastelein

Scholar Week 2016 - present

Presentation Location: Weber Center, Room 101

Abstract

The purpose of this study is to assess the reliability and validity of the 36-item Open Enneagram of Personality Scales (OEPS). Our general hypothesis was that the OEPS would show adequate reliability evidence but not validity evidence. Participants were acquired through a small denominationally affiliated Midwest university, Amazon Mechanical Turk, and social media. Test-retest reliability was done with 249 participants while internal consistency reliability, factor analysis, and correlations with the Big Five Inventory (BFI; John, Donahue, & Kentle) were done using 1039 participants. An average Pearson’s correlation of .68 (range: 0.54 - …

Hidden Framings And Hidden Asymmetries In The Measurement Of Personality: A Combined Lens-Model And Frame-Of-Reference Perspective, Julian Schulze, Stephen G. West, Jan-Philipp Freudenstein, Philipp Schaepers, Patrick Mussel, Michael Eid, Stefan Krumm

Research Collection Lee Kong Chian School Of Business

Objective: The symmetry principle and the frame-of-reference perspective have each made contributions to improving the measurement of personality. Although each perspective is valuable in its own right, we argue that even greater improvement can be achieved through the combination of both. Therefore, the goal of the current article was to show the value of a combined lens-model and frame-of-reference perspective. Method We conducted a literature review to summarize relevant research findings that shed light on the interplay of both perspectives and developed an integrative model. Results: Based on the literature review and on theoretical grounds, we argue that a basic …

Examining The Ecological Validity Of The Power Of Food Scale, Lindsay M. Howard, Kristin E. Heron, Kathryn E. Smith, Ross D. Crosby, Scott G. Engel, Stephen A. Wonderlich, Tyler B. Mason

Psychology Faculty Publications

Purpose

Appetite for palatable foods may impact eating-related behaviors in everyday life. The present study evaluated the real-world predictive validity of the Power of Food Scale (PFS) using ecological momentary assessment (EMA).

Methods

30 women who reported binge eating completed the PFS and related measures. Subsequently, during a 14-day assessment period, participants completed five daily EMA surveys of appetite and binge eating via text message and web.

Results

Results of generalized estimating equations showed that higher PFS scores were associated with higher momentary levels of hunger, eagerness to eat, and urge to eat but were unrelated to fullness, preoccupation with …

A Psychometric Evaluation Of The Best In Class Adherence And Competence Scale, Katrina A. Markowicz

Theses and Dissertations

It is critical to utilize treatment integrity instruments to support the evaluation of evidence-based programs in early classroom contexts. However, in the early childhood field, guidelines for collecting treatment integrity data are underdeveloped. Consequently, most treatment integrity instruments employed in the field solely assess adherence, vary in design features and have little psychometric evidence supporting their use. As such, this represents a gap in the field that might slow efforts to implement evidence-based programs. The current study examines the score reliability and validity of an observational treatment integrity instrument (The BEST in CLASS Adherence and Competence Scale [BiCACS]; Sutherland et …

Adult Attachment Interview Classification: Comparing Two Coding Systems, Patricia M. Hastings

Antioch University Dissertations & Theses

Research on the Adult Attachment Interview (AAI) provides an opportunity to study the ways in which early childhood relational experiences might influence an individual over a lifetime. It is not yet clear, however, whether results from different coding systems for the AAI are equally useful. The first purpose of this study was to compare attachment classification distributions obtained from coding AAIs with the Berkeley and Dynamic-Maturational Model (DMM) coding systems. The second purpose was to explore whether AAI classifications derived from the Berkeley or DMM system were more strongly associated with mother and mother-child dyad outcome variables. Participants were a …

Personnel Selection: A Longstanding Story Of Impact At The Individual, Firm, And Societal Level, Filip Lievens, Paul R. Sackett, Charlene Zhang

Research Collection Lee Kong Chian School Of Business

This paper discusses how and why the field of personnel selection has made a long-lasting mark in work and organizational psychology. We start by outlining the importance and relevance of the well-established analytical framework (criterion-related validity, incremental validity, utility) for examining the impact of selection at the individual (job performance) level. We also document the substantive criterion-related validities of most common selection procedures on the basis of cumulative meta-analytic research. Next, we review more recent research that investigated the impact of selection at the more macro organizational (firm performance) level. We show that the positive relationship between selection and performance …

Development And Initial Validation Of A Brief Questionnaire On The Patients’ View Of The In-Session Realization Of The Six Core Components Of Acceptance And Commitment Therapy, Thomas Probst, Andreas Mühlberger, Johannes Kühner, Georg Eifert, Christoph Pieh, Timo Hackbarth, Johannes Mander

Psychology Faculty Articles and Research

Background: Assessing in-session processes is important in psychotherapy research. The aim of the present study was to create and evaluate a short questionnaire capturing the patients’ view of the in-session realization of the six core components of Acceptance and Commitment Therapy (ACT).

Method: In two studies, psychotherapy patients receiving ACT (Study 1: n = 87) or Cognitive-Behavioral Therapy (CBT) (Study 2, Sample 1: n = 115; Sample 2: n = 156) completed the ACT session questionnaire (ACT-SQ). Therapists were n = 9 ACT therapists (Study 1) and n = 77 CBT trainee therapists (Study 2).

Results: Factor structure: Exploratory factor …

A Psychometric Evaluation Of Computerized Attention Measures In Young Children With Neurofibromatosis Type 1, Sara Katharine Pardej

Theses and Dissertations

Children with neurofibromatosis type 1 (NF1) often demonstrate difficulties with attention and executive functioning that can be evident even starting at a young age. Despite this consistent finding in the literature, there has been no research to determine which measures of attention are most suitable for use with children with NF1. Recently, there have been several computerized measures of attention and executive abilities that have become available to researchers and clinicians. This study explored the National Institute of Health Toolbox Flanker, Dimensional Change Card Sort, and List Sort Working Memory; Cogstate Identification; and Conners Kiddie Continuous Performance Test, second edition …

The Role Of Situations In Situational Judgment Tests: Effects On Construct Saturation, Predictive Validity, And Applicant Perceptions, Philipp Schäpers, Patrick Mussel, Filip Lievens, Cornelius J. König, Jan-Philipp Freudenstein, Stefan Krumm

Research Collection Lee Kong Chian School Of Business

Recent theorizing and empirical evidence suggesting thatsituational judgment tests (SJTs) are more context-independent than previouslythought has sparked a debate about the role of situation descriptions in SJTs.To contribute to this debate and add to our understanding of how SJTs work,this paper conceptually embeds SJT performance in a situation construal modeland examines the effects of situation descriptions on the construct saturationand predictive validity of SJT scores, as well as on applicant perceptions.Across two studies (N = 1,092 and 578) and different SJTs, personality andcognitive ability were equally important determinants of SJT performance regardlessof whether situation descriptions were presented or omitted. The …

Communicating Validity Information To Differentially Experienced Audiences: The Effects Of Numeracy And Nontraditional Metrics, Nathaniel M. Voss, Christopher J. Lake

Personnel Assessment and Decisions

One of the biggest challenges facing organizational researchers is convincing practitioners to adopt evidence-based personnel selection practices such as the structured interview. In this study, we examined the effects of nontraditional validity metrics and numeracy by presenting validity information about the structured interview to audiences with differing amounts of interview experience (students, working adults, and hiring managers). The results indicated that nontraditional metrics were associated with higher understanding, more positive attitudes, and greater perceptions of the usefulness of the structured interview. These effects were constant across differing levels of numeracy. Additionally, the results revealed that nontraditional metrics result in more …

Is It All In The Eye Of The Beholder? The Importance Of Situation Construal For Situational Judgment Test Performance, Jan-Philipp Freudenstein, Philipp Schaepers, Lena Roemer, Patrick Mussel, Stefan Krumm

Research Collection Lee Kong Chian School Of Business

Recent research challenges the importance of situation descriptions for situational judgment test (SJT) performance. This study contributes to resolving the ongoing debate on whether SJTs are situational measures, by incorporating findings on person x situation interactions into SJT research. Specifically, across three studies (N-Total = 1,239), we first tested whether situation construal (i.e., the individual perception of situations in SJTs) predicts responses to SJT items. Second, we assessed whether the relevance of situation construal for SJT performance depends on test elements (i.e., situation descriptions and response options) and item features (i.e., description-dependent vs. description-independent SJT items). Lastly, we determined whether …

Psychometric Developments Of The Ace-Iq: Understanding The Trauma History Of Latine Immigrants, Jorli K. Swingen Dr.

Theses and Dissertations

Adverse childhood experiences (ACEs) have a profound effect on an individual’s physical and mental health. The World Health Organization has recently updated the ACE questionnaire so it could be used with international populations. The Adverse Childhood Experiences-International Questionnaire (ACE-IQ), has not been translated to Spanish or used with Latine immigrants.

This study translated the ACE-IQ into Spanish, evaluated the psychometric properties of the questionnaire, and collected data on 184 four adult English Language Learners in Chesterfield County, VA.

There is evidence of internal consistency for the ACE-IQ as a whole (α = .908) and within subscales. Three factors were identified …

Construct Validity Of The Behavior Assessment System For Children-Third Edition Teacher Rating Scales (Basc-3 Trs): Comparisons With The Adjustment Scales For Children And Adolescents (Asca), Shannon Burback

Masters Theses

The Behavior Assessment Scale for Children-Third Edition Teacher Rating Scale Child Form (BASC-3 TRS-C) and the Adjustment Scales for Children and Adolescents (ASCA) are both teacher rating scales which may be used by school psychologist to assess youth behavior problems. The BASC, BASC-2, and BASC-3 have limited replicated research of the studies reported in their respective manuals. Therefore, it was important to empirically compare the BASC-3 TRS-C with the ASCA to examine construct validity (convergent, discriminant, and divergent) as there were, at present, no published studies replicating BASC-3 Manual research. The present study analyzed BASC-3 TRS-C and ACSA ratings which …