Industrial and Organizational Psychology Commons^™

Multiple, Speeded Assessments Under Scrutiny: Underlying Theory, Design Considerations, Reliability, And Validity, Christoph N. Herde, Filip Lievens

Research Collection Lee Kong Chian School Of Business

Recently, multiple, speeded assessments (e.g., “speeded” or “flash” role-plays) have made rapid inroads into the selection domain. So far, however, the conceptual underpinning and empirical evidence related to these short, fast-paced assessment approaches has been lacking. This raises questions whether these speeded assessments can serve as reliable and valid indicators of future performance. This paper uses the notions of stimulus and response domain sampling to conceptualize multiple, speeded behavioral job simulations as a hybrid of established simulation-based selection methods. Next, we draw upon the thin slices of behavior paradigm to theorize about the quality of ratings made in multiple, speeded …

A Comprehensive Examination Of The Cross-Validity Of Pareto-Optimal Versus Fixed-Weight Selection Systems In The Biobjective Selection Context., Wilfried De Corte, Filip Lievens, Paul R. Sackett

Research Collection Lee Kong Chian School Of Business

The article presents evidence for the cross-validity potential of fixed-weight (FW) versus Pareto-Optimal (PO) selection systems in biobjective selection situations where both the goals of diversity and quality are valued and the importance of the goals is undecided a priori. The article extends previous research by also studying the cross-validity potential of selection systems in the practically most important sample-to-sample cross-validity scenario. We address three research questions: (a) Do different PO systems show comparable levels of relative (i.e., proportional) achievement upon cross-validation? (b) Do PO systems achieve higher levels of relative achievement upon cross-validation than FW selection systems?, and (c) …

Values Assessment For Personnel Selection: Comparing Job Applicants To Non-Applicants, Jeromy Anglim, Karlyn Molloy, Patrick D. Dunlop, Simon L. Albrecht, Filip Lievens, Marty Andrew

Research Collection Lee Kong Chian School Of Business

Some scholars suggest that organizations could improve their hiring decisions by measuring the personal values of job applicants, arguing that values provide insights into applicants’ cultural fit, retention prospects, and performance outcomes. However, others have expressed concerns about response distortion and faking. The current study provides the first large-scale investigation of the effect of the job applicant context on the psychometric structure and scale means of a self-reported values measure. Participants comprised 7,884 job applicants (41% male; age M = 43.32, SD = 10.76) and a country-, age-, and gender-matched comparison sample of 1,806 non-applicants (41% male; age M = …

“Faking” Is Neither Good Nor Bad, It Is A Misleading Concept: A Reply To Tett And Simonet (2021), Bernd Marcus

Personnel Assessment and Decisions

This paper comments on Tett and Simonet’s (2021) outline of two contradictory positions on job applicants’ self-presentation on personality tests labelled “faking is bad” (FIB) versus “faking is good” (FIG). Based on self-presentation theory (Marcus, 2009) Tett and Simonet assigned to their FIG camp, I develop the ideas of (a) understanding self-presentation from the applicant’s rather than the employer’s perspective, (b) avoiding premature moral judgment on this behavior, and (c) examining consequences for the validity of applicant responses with a focus on the intended use for, and the competitive context of, selection. Conclusions include (a) that self-presentation is motivationally and …

Can Gender Pronouns In Interview Questions Work As Nudges?, Fei Lu

Industrial-Organizational Psychology Dissertations

Organizations that are historically male-dominated have struggled to attract and retain an equitable representation of women (Debs et al., 2021; Germain et al., 2012; Hall et al., 2018) Using the two systems processing model from Cognitive Psychology, this study assessed whether gender pronouns can function as environmental cues (“nudges”) to disrupt the patterns of mental models on biases and stereotypes. It was proposed that participants can be “nudged” to decrease the impact of gender stereotype biases in the interview process in male-dominated professions (e.g., Information Technology) such that pronouns used in the interview questions will interact with the interviewee’s gender. …

Scientific, Legal, And Ethical Concerns About Ai-Based Personnel Selection Tools: A Call To Action, Nancy T. Tippins, Frederick L. Oswald, S. Morton Mcphail

Personnel Assessment and Decisions

Organizations are increasingly turning toward personnel selection tools that rely on artificial intelligence (AI) technologies and machine learning algorithms that, together, intend to predict the future success of employees better than traditional tools. These new forms of assessment include online games, video-based interviews, and big data pulled from many sources, including test responses, test-taking behavior, applications, resumes, and social media. Speedy processing, lower costs, convenient access, and applicant engagement are often and rightfully cited as the practical advantages for using these selection tools. At the same time, however, these tools raise serious concerns about their effectiveness in terms of their …

Applicant Faking On Personality Tests: Good Or Bad And Why Should We Care?, Robert P. Tett, Daniel V. Simonet

Personnel Assessment and Decisions

The unitarian understanding of construct validity holds that deliberate response distortion in completing self-report personality tests (i.e., faking) threatens trait-based inferences drawn from test scores. This “faking-is-bad” (FIB) perspective is being challenged by an emerging “faking-is-good” (FIG) position that condones or favors faking and its underlying attributes (e.g., social skill, ATIC) to the degree they contribute to predictor–criterion correlations and are job relevant. Based on the unitarian model of validity and relevant empirical evidence, we argue the FIG perspective is psychometrically flawed and counterproductive to personality-based selection targeting trait-based fit. Carrying forward both positions leads to variously dark futures for …

Faking And The Validity Of Personality Tests: An Experimental Investigation Using Modern Forced Choice Measures, Christopher R. Huber, Nathan R. Kuncel, Katie B. Huber, Anthony S. Boyce

Personnel Assessment and Decisions

Despite the established validity of personality measures for personnel selection, their susceptibility to faking has been a persistent concern. However, the lack of studies that combine generalizability with experimental control makes it difficult to determine the effects of applicant faking. This study addressed this deficit in two ways. First, we compared a subtle incentive to fake with the explicit “fake-good” instructions used in most faking experiments. Second, we compared standard Likert scales to multidimensional forced choice (MFC) scales designed to resist deception, including more and less fakable versions of the same MFC inventory. MFC scales substantially reduced motivated score elevation …

What's On Job Seekers' Social Media Sites? A Content Analysis And Effects Of Structure On Recruiter Judgments And Predictive Validity, Liwen Zhang, Chad H. Van Iddekinge, John D. Arnold, Philip L. Roth, Filip Lievens, Stephen E. Lanivich, Samantha L. Jordan

Research Collection Lee Kong Chian School Of Business

Many organizational representatives review social media (SM) information (e.g., Facebook, Twitter) when recruiting and assessing job applicants. Despite this, very little empirical data exist concerning the SM information available to organizations or whether assessments of such information are a valid predictor of work outcomes. This multi-study investigation examines several critical issues in this emerging area. In Study 1, we conducted a content analysis of job seekers’ Facebook sites (n = 266) and found that these sites often provide demographic variables that U.S. employment laws typically prohibit organizations from using when making personnel decisions (e.g., age, ethnicity, religion), as well as …

Threat Of Technological Unemployment, Use Intentions, And The Promotion Of Structured Interviews In Personnel Selection, Kevin P. Nolan, Dev K. Dalal, Nathan Carter

Personnel Assessment and Decisions

Meehl (1986) proposed that an important factor underlying professional decision makers’ resistance to standardized decision aids is threat of technological unemployment – fear that using the practices would reduce the perceived value of their employment. Nolan, Carter, and Dalal (2016) provided initial support for threat of technological unemployment being a factor that contributes to practitioners’ reluctance to adopt scientifically meritorious standardized hiring practices. This study serves to further develop the theory of threat of technological unemployment in personnel selection by (a) replicating the findings of our earlier research using a within-subjects methodology that is more generalizable to the cognitive processes …

Decoy Effects Improve Diversity Hiring, Nathan R. Kuncel, Jeffrey A. Dahlke

Personnel Assessment and Decisions

A growing literature demonstrates that when making choices among multiple options, decision makers are strongly influenced by the mere presence of additional options, even when those options are largely undesirable and are never actually selected. The effects of irrelevant options on decisions, often called decoy effects, have been observed in hiring and admissions decisions where the nature of a third candidate can radically shift preferences. In this study, we examine the influence of decoy effects on diversity hiring and extend research by examining choices with more than two organizational goals. Results indicate that the presence of a second candidate who …

Does Feedback Increase Decision Aid Use Among Hiring Professionals?, Aneeqa Thiele, Alexander T. Jackson, Stacey M. Stremic, Satoris S. Howes

Personnel Assessment and Decisions

We examined the influence of formative and outcome feedback on people’s reliance on decision aids. Decision aids are tools that managers can use to increase the accuracy of their hiring decisions. In our study, participants were asked to make 20 different hiring decisions and make predictions of a candidate’s performance on the job, with the option of using a decision aid formula. We manipulated whether participants received feedback on the accuracy of their predictions, the accuracy of the decision aid’s predictions, or both. The results demonstrated that feedback failed to have a significant impact on decision aid use for both …

Pushing The Limits For Judgmental Consistency: Comparing Random Weighting Schemes With Expert Judgments, Martin C. Yu, Nathan R. Kuncel

Personnel Assessment and Decisions

Consistent use of information has been identified as a critical issue that can undermine expert predictions. Using three personnel assessment datasets, we conduct Monte Carlo simulations to compare the accuracy of expert judgements for predicting the job performance of managers against four different weighting schemes: consistent random weights, completely random weights, unit weights, and optimal weights. Expert accuracy fell within the completely random weight distribution in two samples and at the low end of the consistent random weight distribution in one sample. In other words, consistent random weights reliably outperformed expert judgment for hiring decisions across three datasets with a …

Multiple Speed Assessments: Theory, Practice, And Research Evidence, Christoph N. Herde, Filip Lievens

Research Collection Lee Kong Chian School Of Business

This paper presents Multiple Speed Assessments as an umbrella term to encompass a variety of approaches that include multiple (e.g., 20), short (e.g., 3 min), and often integrated interpersonal simulations to elicit overt behavior in a standardized way across participants. Multiple Speed Assessments can be used to get insight into the behavioral repertoire of a target person in situations sampled from a predefined target domain and their intraindividual variability across these situations. This paper outlines the characteristics and theoretical basis of Multiple Speed Assessments. We also discuss various already existing examples of Multiple Speed Assessments (Objective Structured Clinical Examinations, Multiple …

“Where’S The I-O?” Artificial Intelligence And Machine Learning In Talent Management Systems, Manuel F. Gonzalez, John F. Capman, Frederick L. Oswald, Evan R. Theys, David L. Tomczak

Personnel Assessment and Decisions

Artificial intelligence (AI) and machine learning (ML) have seen widespread adoption by organizations seeking to identify and hire high-quality job applicants. Yet the volume, variety, and velocity of professional involvement among I-O psychologists remains relatively limited when it comes to developing and evaluating AI/ML applications for talent assessment and selection. Furthermore, there is a paucity of empirical research that investigates the reliability, validity, and fairness of AI/ML tools in organizational contexts. To stimulate future involvement and research, we share our review and perspective on the current state of AI/ML in talent assessment as well as its benefits and potential pitfalls; …

A New Scoring Procedure In Assessment Centers: Insights From Interaction Analysis, Janneke K. Oostrom, Nale Lehmann-Willenbrock, Ute-Christine Klehe

Personnel Assessment and Decisions

This paper proposes interaction analysis as an alternative scoring procedure in assessment centers (ACs). Interaction analysis allows for a more fine-grained scoring approach by which candidate behaviors are captured as they actually happen, thus avoiding judgment errors typically associated with traditional scoring procedures. We describe interaction analysis and explain how this procedure can improve the validity of ACs. In a short research example, we showcase how interaction analysis can be implemented in AC settings. Finally, we integrate our arguments in terms of three key propositions which we hope will inspire future research on more dynamic scoring procedures.

Selection Tool Use: A Focus On Personality Testing In Canada, The United States, And Germany, Stephen D. Risavy, Peter A. Fisher, Chet Robie, Cornelius J. König

Personnel Assessment and Decisions

The purpose of this paper is to provide new data regarding the current staffing practices being used by organizations in Canada and the United States (US) as well as a comparison with existing data from Germany (Diekmann & König, 2015). Data regarding the beliefs of human resource (HR) practitioners in terms of using personality tests in personnel selection is also provided. A geographically representative sample of 453 HR practitioners across Canada and the US were surveyed. Although general mental ability testing has previously been found to be highly valid and cost effective, this selection tool was among the least commonly …

Examining Factors Influencing Use Of A Decision Aid In Personnel Selection, Alexander T. Jackson, Michael E. Young, Satoris S. Howes, Patrick A. Knight, Sydney L. Reichin

Personnel Assessment and Decisions

In this research, two studies were conducted to examine factors influencing reliance on a decision aid in personnel selection. Specifically, this study examined the effect of feedback, validity of selection predictors, and presence of a decision aid on the use of the aid in personnel selection. The results demonstrate that when people are provided with the decision aid, their predictions were significantly more similar to the predictions made by the aid than people who were not provided with the aid. This suggests that when people are provided with an aid, they will use it to some degree. This research also …

Ecological Rationality: Fast-And-Frugal Heuristics For Managerial Decision Making Under Uncertainty, Shenghua Luan, Jochen Reb, Gerd Gigerenzer

Research Collection Lee Kong Chian School Of Business

Heuristics are often viewed as inferior to “rational” strategies that exhaustively search and process information. Introducing the theoretical perspective of ecological rationality, we challenge this view and argue that under conditions of uncertainty common to managerial decision making, managers can actually make better decisions using fast-and-frugal heuristics. Within the context of personnel selection, we show that a heuristic called Δ-inference can more accurately predict which of two job applicants would perform better in the future than logistic regression, a prototypical rational strategy. Using data from 236 applicants at an airline company, we demonstrate in Study 1 that despite searching less …

The Effects Of A Blind Selection Process On Gender Discrimination In Applicant Selection, Stephanie Ann Ingalls

Electronic Theses, Projects, and Dissertations

The purpose of the current study was to examine the effect of a blind selection process on gender discrimination. Due to persistent gender discrimination in selection processes, the intention of the current study was to investigate a blind selection process as a means to decrease gender discrimination against women. A total of 391 individuals were recruited through SONA and convenience sampling to participate in the current study. Materials included a selection scenario, three applicant résumés with applicant names and three with applicant ID numbers, a rank order form, and measures for procedural justice and fairness, modern sexism inventory, and the …

Construct-Driven Sjts: Toward An Agenda For Future Research, Filip Lievens

Research Collection Lee Kong Chian School Of Business

A common theme running through recent research on Situational Judgment Tests (SJTs) and this special issue is the aim to improve the measurement of constructs via SJTs. Construct-driven SJTs differ from traditional SJTs in that they present a trait activating situation to test-takers and a more unidimensional set of response options that depict different trait levels. In this commentary, I frame the different papers of this special issue into a research agenda related to construct-driven SJTs. Specifically, I posit that future research should examine whether construct-driven SJTs lead to more unidimensionality at the item level, cleaner measurement of the constructs, …

Understanding The Building Blocks Of Selection Procedures: Effects Of Response Fidelity On Performance And Validity, Filip Lievens, Wilfried De Corte, Lena Westerveld

Research Collection Lee Kong Chian School Of Business

This study aims to advance our conceptual understanding of selection procedures by exploring the effect of response fidelity (i.e., written constructed response vs. behavioral constructed response) on test performance, validity, and applicant perceptions. Stimulus fidelity (multimedia stimulus) was kept constant. In a field experiment, 208 applicants for entry-level police officer jobs completed a multimedia situational judgment test with written constructed responses and behavioral responses. We hypothesized the behavioral response mode (a) to be a better predictor of police trainee performance one year later, (b) to be less cognitively saturated, (c) to exhibit higher personality (extraversion) saturation, and (d) to be …

Personnel Selection, Safety Performance, And Job Performance: Are Safe Workers Better Workers?, Erica N. Drew

FIU Electronic Theses and Dissertations

The present dissertation consists of two studies that combine personnel selection, safety performance, and job performance literatures to answer an important question: are safe workers better workers? Study 1 tested a predictive model of safety performance to examine personality characteristics (conscientiousness and agreeableness), and two novel behavioral constructs (safety orientation and safety judgment) as predictors of safety performance in a sample of forklift loaders/operators (N = 307). Analyses centered on investigating safety orientation as a proximal predictor and determinant of safety performance. Study 2 replicated Study 1 and explored the relationship between safety performance and job performance by testing an …

Do Candidate Reactions Relate To Job Performance Or Affect Criterion-Related Validity? A Multistudy Investigation Of Relations Among Reactions, Selection Test Scores, And Job Performance, Julie Mccarthy, Chad H. Van Iddekinge, Filip Lievens, Mavis Mei-Chuan Kung, Evan F. Sinar, Michael A. Campion

Research Collection Lee Kong Chian School Of Business

Considerable evidence suggests that how candidates react to selection procedures can affect their test performance and their attitudes toward the hiring organization (e.g., recommending the firm to others). However, very few studies of candidate reactions have examined one of the outcomes organizations care most about: job performance. We attempt to address this gap by developing and testing a conceptual framework that delineates whether and how candidate reactions might influence job performance. We accomplish this objective using data from 4 studies (total N = 6,480), 6 selection procedures (personality tests, job knowledge tests, cognitive ability tests, work samples, situational judgment tests, …

The Life Individual Resilience Scale And Cognitive Reflection Test : Predictive Utility In Special Forces Selection, Anthony Moffitt

Theses : Honours

The use of psychometric testing by the military in the screening and selection of its personnel has played a significant role in test development (Aiken, 1997). Several studies have demonstrated the utility of such tests to identify and select elite Special Forces personnel specifically for their employment in complex and dangerous environments (Picano, Roland, Williams, & Rollins, 2006). Research of additional discrete personal attributes beyond traditional cognitive ability and personality traits has inspired studies into the measurement of specific aspects of performance, including perseverance, hardiness and decision-making (e.g., Beal, 2010; Temby & Drobnjak, 2010). However, the empirical evidence for these …

Designing Pareto-Optimal Selection Systems: Formalizing The Decisions Required For Selection System Development, Wilfried De Corte, Paul R. Sackett, Filip Lievens

Research Collection Lee Kong Chian School Of Business

The article presents an analytic method for designing Pareto-optimal selection systems where the applicants belong to a mixture of candidate populations. The method is useful in both applied and research settings. In an applied context, the present method is the first to assist the selection practitioner when deciding on 6 major selection design issues: (1) the predictor subset, (2) the selection rule, (3) the selection staging, (4) the predictor sequencing, (5) the predictor weighting, and (6) the stage retention decision issue. From a research perspective, the method offers a unique opportunity for studying the impact and relative importance of different …

Exploring Process Dissociation As A Tool For Investigating Discrimination In Hiring Situations, Rhys J. Lewis

Electronic Thesis and Dissertation Repository

Process dissociation is introduced as a way to overcome methodological limitations currently hindering sexism research. Researchers have identified two main types of sexism in hiring contexts. Meta-analyses confirm that men are traditionally advantaged over women (Tosi & Einbender, 1985), and that both genders encounter discrimination when applying to a job typically associated with the other gender (Davison & Burke, 2000). One problem is that these two biases are often confounded. As a result, researchers have hitherto been limited to showing that the two biases exist, but are largely unable to quantify them.

A possible solution might be process dissociation. It …

Applicant Versus Employee Scores On Self-Report Emotional Intelligence Measures, Filip Lievens, Ute-Christine Klehe, Nele Libbrecht

Research Collection Lee Kong Chian School Of Business

There exists growing interest to assess applicants' emotional intelligence (EI) via self-report trait-based measures of EI as part of the selection process. However, some studies that experimentally manipulated applicant conditions have cautioned that in these conditions use of self-report measures for assessing EI might lead to considerably higher scores than current norm scores suggest. So far, no studies have scrutinized self-reported EI scores among a sample of actual job applicants. Therefore, this study compares the scores of actual applicants at a large ICT organization (n = 109) on a well-known self-report measure of EI to the scores of employees already …

Dimension And Exercise Variance In Assessment Center Scores: A Large-Scale Evaluation Of Multitrait-Multimethod Studies, Filip Lievens, James M. Conway

Research Collection Lee Kong Chian School Of Business

This study addresses 3 questions regarding assessment center construct validity: (a) Are assessment center ratings best thought of is reflecting dimension constructs (dimension model). exercises (exercise model). or a combination? (b) To what extent do dimensions or exercises account for variance? (c) Which design characteristics increase dimension variance? To this end, a large set of multitrait-multimethod studies (N = 34) were analyzed, showing that assessment center ratings were best represented (i.e., in terms of fit and admissible solutions) by a model with correlated dimensions and exercises specified a correlated uniquenesses. In this model, dimension variance equals exercise variance. Significantly more …