Business Commons^™

Multiple, Speeded Assessments Under Scrutiny: Underlying Theory, Design Considerations, Reliability, And Validity, Christoph N. Herde, Filip Lievens

Research Collection Lee Kong Chian School Of Business

Recently, multiple, speeded assessments (e.g., “speeded” or “flash” role-plays) have made rapid inroads into the selection domain. So far, however, the conceptual underpinning and empirical evidence related to these short, fast-paced assessment approaches has been lacking. This raises questions whether these speeded assessments can serve as reliable and valid indicators of future performance. This paper uses the notions of stimulus and response domain sampling to conceptualize multiple, speeded behavioral job simulations as a hybrid of established simulation-based selection methods. Next, we draw upon the thin slices of behavior paradigm to theorize about the quality of ratings made in multiple, speeded …

A Comprehensive Examination Of The Cross-Validity Of Pareto-Optimal Versus Fixed-Weight Selection Systems In The Biobjective Selection Context., Wilfried De Corte, Filip Lievens, Paul R. Sackett

Research Collection Lee Kong Chian School Of Business

The article presents evidence for the cross-validity potential of fixed-weight (FW) versus Pareto-Optimal (PO) selection systems in biobjective selection situations where both the goals of diversity and quality are valued and the importance of the goals is undecided a priori. The article extends previous research by also studying the cross-validity potential of selection systems in the practically most important sample-to-sample cross-validity scenario. We address three research questions: (a) Do different PO systems show comparable levels of relative (i.e., proportional) achievement upon cross-validation? (b) Do PO systems achieve higher levels of relative achievement upon cross-validation than FW selection systems?, and (c) …

Values Assessment For Personnel Selection: Comparing Job Applicants To Non-Applicants, Jeromy Anglim, Karlyn Molloy, Patrick D. Dunlop, Simon L. Albrecht, Filip Lievens, Marty Andrew

Research Collection Lee Kong Chian School Of Business

Some scholars suggest that organizations could improve their hiring decisions by measuring the personal values of job applicants, arguing that values provide insights into applicants’ cultural fit, retention prospects, and performance outcomes. However, others have expressed concerns about response distortion and faking. The current study provides the first large-scale investigation of the effect of the job applicant context on the psychometric structure and scale means of a self-reported values measure. Participants comprised 7,884 job applicants (41% male; age M = 43.32, SD = 10.76) and a country-, age-, and gender-matched comparison sample of 1,806 non-applicants (41% male; age M = …

“Faking” Is Neither Good Nor Bad, It Is A Misleading Concept: A Reply To Tett And Simonet (2021), Bernd Marcus

Personnel Assessment and Decisions

This paper comments on Tett and Simonet’s (2021) outline of two contradictory positions on job applicants’ self-presentation on personality tests labelled “faking is bad” (FIB) versus “faking is good” (FIG). Based on self-presentation theory (Marcus, 2009) Tett and Simonet assigned to their FIG camp, I develop the ideas of (a) understanding self-presentation from the applicant’s rather than the employer’s perspective, (b) avoiding premature moral judgment on this behavior, and (c) examining consequences for the validity of applicant responses with a focus on the intended use for, and the competitive context of, selection. Conclusions include (a) that self-presentation is motivationally and …

Smart Heuristics For Individuals, Teams, And Organizations, Gerd Gigerenzer, Jochen Reb, Shenghua Luan

Research Collection Lee Kong Chian School Of Business

Heuristics are fast, frugal, and accurate strategies that enable rather than limit decision making under uncertainty. Uncertainty, as opposed to calculable risk, is characteristic of most organizational contexts. We review existing research and offer a descriptive and prescriptive theoretical framework to integrate the current patchwork of heuristics scattered across various areas of organizational studies. Research on the adaptive toolbox is descriptive, identifying the repertoire of heuristics on which individuals, teams, and organizations rely. Research on ecological rationality is prescriptive, specifying the conditions under which a given heuristic performs well, that is, when it is smart. Our review finds a relatively …

Scientific, Legal, And Ethical Concerns About Ai-Based Personnel Selection Tools: A Call To Action, Nancy T. Tippins, Frederick L. Oswald, S. Morton Mcphail

Personnel Assessment and Decisions

Organizations are increasingly turning toward personnel selection tools that rely on artificial intelligence (AI) technologies and machine learning algorithms that, together, intend to predict the future success of employees better than traditional tools. These new forms of assessment include online games, video-based interviews, and big data pulled from many sources, including test responses, test-taking behavior, applications, resumes, and social media. Speedy processing, lower costs, convenient access, and applicant engagement are often and rightfully cited as the practical advantages for using these selection tools. At the same time, however, these tools raise serious concerns about their effectiveness in terms of their …

Applicant Faking On Personality Tests: Good Or Bad And Why Should We Care?, Robert P. Tett, Daniel V. Simonet

Personnel Assessment and Decisions

The unitarian understanding of construct validity holds that deliberate response distortion in completing self-report personality tests (i.e., faking) threatens trait-based inferences drawn from test scores. This “faking-is-bad” (FIB) perspective is being challenged by an emerging “faking-is-good” (FIG) position that condones or favors faking and its underlying attributes (e.g., social skill, ATIC) to the degree they contribute to predictor–criterion correlations and are job relevant. Based on the unitarian model of validity and relevant empirical evidence, we argue the FIG perspective is psychometrically flawed and counterproductive to personality-based selection targeting trait-based fit. Carrying forward both positions leads to variously dark futures for …

Faking And The Validity Of Personality Tests: An Experimental Investigation Using Modern Forced Choice Measures, Christopher R. Huber, Nathan R. Kuncel, Katie B. Huber, Anthony S. Boyce

Personnel Assessment and Decisions

Despite the established validity of personality measures for personnel selection, their susceptibility to faking has been a persistent concern. However, the lack of studies that combine generalizability with experimental control makes it difficult to determine the effects of applicant faking. This study addressed this deficit in two ways. First, we compared a subtle incentive to fake with the explicit “fake-good” instructions used in most faking experiments. Second, we compared standard Likert scales to multidimensional forced choice (MFC) scales designed to resist deception, including more and less fakable versions of the same MFC inventory. MFC scales substantially reduced motivated score elevation …

What's On Job Seekers' Social Media Sites? A Content Analysis And Effects Of Structure On Recruiter Judgments And Predictive Validity, Liwen Zhang, Chad H. Van Iddekinge, John D. Arnold, Philip L. Roth, Filip Lievens, Stephen E. Lanivich, Samantha L. Jordan

Research Collection Lee Kong Chian School Of Business

Many organizational representatives review social media (SM) information (e.g., Facebook, Twitter) when recruiting and assessing job applicants. Despite this, very little empirical data exist concerning the SM information available to organizations or whether assessments of such information are a valid predictor of work outcomes. This multi-study investigation examines several critical issues in this emerging area. In Study 1, we conducted a content analysis of job seekers’ Facebook sites (n = 266) and found that these sites often provide demographic variables that U.S. employment laws typically prohibit organizations from using when making personnel decisions (e.g., age, ethnicity, religion), as well as …

Little Did We Know What We Are Preparing For: Virtual Interviews Before Covid-19, Alina Popa, Scott Kubomoto, Lorra Baldes

HCA Healthcare Journal of Medicine

Current circumstances related to the COVID-19 pandemic require residency programs to conduct virtual interviews. We draw upon our experience with virtual interviews from fall 2019, prior to COVID-19, and propose a potential implementation process for virtual interviewing strategies. We also describe potential pitfalls and strategies to mitigate them.

Threat Of Technological Unemployment, Use Intentions, And The Promotion Of Structured Interviews In Personnel Selection, Kevin P. Nolan, Dev K. Dalal, Nathan Carter

Personnel Assessment and Decisions

Meehl (1986) proposed that an important factor underlying professional decision makers’ resistance to standardized decision aids is threat of technological unemployment – fear that using the practices would reduce the perceived value of their employment. Nolan, Carter, and Dalal (2016) provided initial support for threat of technological unemployment being a factor that contributes to practitioners’ reluctance to adopt scientifically meritorious standardized hiring practices. This study serves to further develop the theory of threat of technological unemployment in personnel selection by (a) replicating the findings of our earlier research using a within-subjects methodology that is more generalizable to the cognitive processes …

Decoy Effects Improve Diversity Hiring, Nathan R. Kuncel, Jeffrey A. Dahlke

Personnel Assessment and Decisions

A growing literature demonstrates that when making choices among multiple options, decision makers are strongly influenced by the mere presence of additional options, even when those options are largely undesirable and are never actually selected. The effects of irrelevant options on decisions, often called decoy effects, have been observed in hiring and admissions decisions where the nature of a third candidate can radically shift preferences. In this study, we examine the influence of decoy effects on diversity hiring and extend research by examining choices with more than two organizational goals. Results indicate that the presence of a second candidate who …

Does Feedback Increase Decision Aid Use Among Hiring Professionals?, Aneeqa Thiele, Alexander T. Jackson, Stacey M. Stremic, Satoris S. Howes

Personnel Assessment and Decisions

We examined the influence of formative and outcome feedback on people’s reliance on decision aids. Decision aids are tools that managers can use to increase the accuracy of their hiring decisions. In our study, participants were asked to make 20 different hiring decisions and make predictions of a candidate’s performance on the job, with the option of using a decision aid formula. We manipulated whether participants received feedback on the accuracy of their predictions, the accuracy of the decision aid’s predictions, or both. The results demonstrated that feedback failed to have a significant impact on decision aid use for both …

Pushing The Limits For Judgmental Consistency: Comparing Random Weighting Schemes With Expert Judgments, Martin C. Yu, Nathan R. Kuncel

Personnel Assessment and Decisions

Consistent use of information has been identified as a critical issue that can undermine expert predictions. Using three personnel assessment datasets, we conduct Monte Carlo simulations to compare the accuracy of expert judgements for predicting the job performance of managers against four different weighting schemes: consistent random weights, completely random weights, unit weights, and optimal weights. Expert accuracy fell within the completely random weight distribution in two samples and at the low end of the consistent random weight distribution in one sample. In other words, consistent random weights reliably outperformed expert judgment for hiring decisions across three datasets with a …

Robustness, Sensitivity And Sampling Variability Of Pareto-Optimal Selection System Solutions To Address The Quality-Diversity Trade-Off, Wilfried De Corte, Paul Sackett, Filip Lievens

Research Collection Lee Kong Chian School Of Business

In case that both the goals of selection quality and diversity are important, a selection system is Pareto-optimal (PO) when its implementation is expected to result in an optimal balance between the levels achieved with respect to both these goals. The study addresses the critical issue whether PO systems, as computed from calibration conditions, continue to perform well when applied to a large variety of different validation selection situations. To address the key issue, we introduce two new measures for gauging the achievement of these designs and conduct a large simulation study in which we manipulate 10 factors (related to …

Multiple Speed Assessments: Theory, Practice, And Research Evidence, Christoph N. Herde, Filip Lievens

Research Collection Lee Kong Chian School Of Business

This paper presents Multiple Speed Assessments as an umbrella term to encompass a variety of approaches that include multiple (e.g., 20), short (e.g., 3 min), and often integrated interpersonal simulations to elicit overt behavior in a standardized way across participants. Multiple Speed Assessments can be used to get insight into the behavioral repertoire of a target person in situations sampled from a predefined target domain and their intraindividual variability across these situations. This paper outlines the characteristics and theoretical basis of Multiple Speed Assessments. We also discuss various already existing examples of Multiple Speed Assessments (Objective Structured Clinical Examinations, Multiple …

A Meta-Analysis Of E-Recruitment Applicant Experience, Perception, And Behavior, Diem J. Mooney

Walden Dissertations and Doctoral Studies

E-recruitment, an Internet-based approach for recruitment, has been shown to be an efficient method for organizations to reach a target candidate population. However, challenges with recruitment websites have resulted in high rates of job seekers abandoning an online application before completion. A quantitative meta-analytic review of literature from 2009 to 2019 was conducted to determine the relationship between applicant user experience (UX) when interacting with an organization’s e-recruitment website, applicant perceptions of the hiring organization, and applicant behaviors related to pursuing employment. The review initially considered 105 candidate articles, of which 41 were consider after applying initial inclusions criteria. After …

“Where’S The I-O?” Artificial Intelligence And Machine Learning In Talent Management Systems, Manuel F. Gonzalez, John F. Capman, Frederick L. Oswald, Evan R. Theys, David L. Tomczak

Personnel Assessment and Decisions

Artificial intelligence (AI) and machine learning (ML) have seen widespread adoption by organizations seeking to identify and hire high-quality job applicants. Yet the volume, variety, and velocity of professional involvement among I-O psychologists remains relatively limited when it comes to developing and evaluating AI/ML applications for talent assessment and selection. Furthermore, there is a paucity of empirical research that investigates the reliability, validity, and fairness of AI/ML tools in organizational contexts. To stimulate future involvement and research, we share our review and perspective on the current state of AI/ML in talent assessment as well as its benefits and potential pitfalls; …

A New Scoring Procedure In Assessment Centers: Insights From Interaction Analysis, Janneke K. Oostrom, Nale Lehmann-Willenbrock, Ute-Christine Klehe

Personnel Assessment and Decisions

This paper proposes interaction analysis as an alternative scoring procedure in assessment centers (ACs). Interaction analysis allows for a more fine-grained scoring approach by which candidate behaviors are captured as they actually happen, thus avoiding judgment errors typically associated with traditional scoring procedures. We describe interaction analysis and explain how this procedure can improve the validity of ACs. In a short research example, we showcase how interaction analysis can be implemented in AC settings. Finally, we integrate our arguments in terms of three key propositions which we hope will inspire future research on more dynamic scoring procedures.

Selection Tool Use: A Focus On Personality Testing In Canada, The United States, And Germany, Stephen D. Risavy, Peter A. Fisher, Chet Robie, Cornelius J. König

Personnel Assessment and Decisions

The purpose of this paper is to provide new data regarding the current staffing practices being used by organizations in Canada and the United States (US) as well as a comparison with existing data from Germany (Diekmann & König, 2015). Data regarding the beliefs of human resource (HR) practitioners in terms of using personality tests in personnel selection is also provided. A geographically representative sample of 453 HR practitioners across Canada and the US were surveyed. Although general mental ability testing has previously been found to be highly valid and cost effective, this selection tool was among the least commonly …

Examining Factors Influencing Use Of A Decision Aid In Personnel Selection, Alexander T. Jackson, Michael E. Young, Satoris S. Howes, Patrick A. Knight, Sydney L. Reichin

Personnel Assessment and Decisions

In this research, two studies were conducted to examine factors influencing reliance on a decision aid in personnel selection. Specifically, this study examined the effect of feedback, validity of selection predictors, and presence of a decision aid on the use of the aid in personnel selection. The results demonstrate that when people are provided with the decision aid, their predictions were significantly more similar to the predictions made by the aid than people who were not provided with the aid. This suggests that when people are provided with an aid, they will use it to some degree. This research also …

Ecological Rationality: Fast-And-Frugal Heuristics For Managerial Decision Making Under Uncertainty, Shenghua Luan, Jochen Reb, Gerd Gigerenzer

Research Collection Lee Kong Chian School Of Business

Heuristics are often viewed as inferior to “rational” strategies that exhaustively search and process information. Introducing the theoretical perspective of ecological rationality, we challenge this view and argue that under conditions of uncertainty common to managerial decision making, managers can actually make better decisions using fast-and-frugal heuristics. Within the context of personnel selection, we show that a heuristic called Δ-inference can more accurately predict which of two job applicants would perform better in the future than logistic regression, a prototypical rational strategy. Using data from 236 applicants at an airline company, we demonstrate in Study 1 that despite searching less …

Assessing Personality Dynamics In Personnel Selection, Joanna Sosnowska, Joeri Hofmans, Filip Lievens

Research Collection Lee Kong Chian School Of Business

Recently, there have been repeated calls in the literature for an integrative approach to personality, in which both between- and within-person fluctuations are simultaneously considered. Although the integrative approach to personality offers a compelling extension of the traditional trait approach, one of the major challenges is its applicability in applied settings. In the present chapter, we address this challenge for the domain of personnel selection, showing that an integrative approach to personality assessment in selection settings is possible through careful consideration of available theories and selection methods. By explaining and delineating how existing concepts can be used and how existing …

Construct-Driven Sjts: Toward An Agenda For Future Research, Filip Lievens

Research Collection Lee Kong Chian School Of Business

A common theme running through recent research on Situational Judgment Tests (SJTs) and this special issue is the aim to improve the measurement of constructs via SJTs. Construct-driven SJTs differ from traditional SJTs in that they present a trait activating situation to test-takers and a more unidimensional set of response options that depict different trait levels. In this commentary, I frame the different papers of this special issue into a research agenda related to construct-driven SJTs. Specifically, I posit that future research should examine whether construct-driven SJTs lead to more unidimensionality at the item level, cleaner measurement of the constructs, …

The Effects Of Predictor Method Factors On Selection Outcomes: A Modular Approach To Personnel Selection Procedures, Filip Lievens, Paul R. Sackett

Research Collection Lee Kong Chian School Of Business

Past reviews and meta-analyses typically conceptualized and examined selection procedures as holistic entities. We draw on the product design literature to propose a modular approach as a complementary perspective to conceptualizing selection procedures. A modular approach means that a product is broken down into its key underlying components. Therefore, we start by presenting a modular framework that identifies the important measurement components of selection procedures. Next, we adopt this modular lens for reviewing the available evidence regarding each of these components in terms of affecting validity, subgroup differences, and applicant perceptions, as well as for identifying new research directions. As …

The Cross-National Generalizability Of Biographical Data: An Examination Within A Multinational Organization, Adam J. Ducey

USF Tampa Graduate Theses and Dissertations

In an increasingly interconnected economy, organizations are frequently operating beyond national borders. International partnerships, joint ventures, mergers, and acquisitions have expanded the labor market from a domestic to an international perspective. In this environment, multinational organizations demand cost-effective personnel selection tools to help them identify top talent from different countries, geographical regions, and cultures.

The purpose of the current research was to evaluate the global utility of biographical data inventories, a standardized self-report selection method that asks job applicants questions about prior behaviors and experiences. Results from two studies involving participants from 7 country clusters, across four continents, and two …

Understanding The Building Blocks Of Selection Procedures: Effects Of Response Fidelity On Performance And Validity, Filip Lievens, Wilfried De Corte, Lena Westerveld

Research Collection Lee Kong Chian School Of Business

This study aims to advance our conceptual understanding of selection procedures by exploring the effect of response fidelity (i.e., written constructed response vs. behavioral constructed response) on test performance, validity, and applicant perceptions. Stimulus fidelity (multimedia stimulus) was kept constant. In a field experiment, 208 applicants for entry-level police officer jobs completed a multimedia situational judgment test with written constructed responses and behavioral responses. We hypothesized the behavioral response mode (a) to be a better predictor of police trainee performance one year later, (b) to be less cognitively saturated, (c) to exhibit higher personality (extraversion) saturation, and (d) to be …

The Ethical Implications Of Using Genetic Information In Personnel Selection, Brent B. Clark, Chet E. Barney, Tyler Reddington

Marketing and Management Faculty Publications

Biology, during the last decade in particular, is making substantial headway into our social theories of business and behavior. While the social sciences rush to keep up with the advancement of knowledge, we highlight the need for an ethics discussion to also keep pace. Although the implications to theory are important, our focus is on how new knowledge has the capacity to alter the formulation and practice of business policy, which we believe is potentially profound. Furthermore, the ethicality of a set of issues can depend heavily on one’s perspective, and differing views may not always be compatible. With this …

Do Candidate Reactions Relate To Job Performance Or Affect Criterion-Related Validity? A Multistudy Investigation Of Relations Among Reactions, Selection Test Scores, And Job Performance, Julie Mccarthy, Chad H. Van Iddekinge, Filip Lievens, Mavis Mei-Chuan Kung, Evan F. Sinar, Michael A. Campion

Research Collection Lee Kong Chian School Of Business

Considerable evidence suggests that how candidates react to selection procedures can affect their test performance and their attitudes toward the hiring organization (e.g., recommending the firm to others). However, very few studies of candidate reactions have examined one of the outcomes organizations care most about: job performance. We attempt to address this gap by developing and testing a conceptual framework that delineates whether and how candidate reactions might influence job performance. We accomplish this objective using data from 4 studies (total N = 6,480), 6 selection procedures (personality tests, job knowledge tests, cognitive ability tests, work samples, situational judgment tests, …

Responding To Personality Tests In A Selection Context: The Role Of The Ability To Identify Criteria And The Ideal-Employee Factor, Ute-Christine Kelhe, Martin Kleinmann, Thomas Hartstein, Klaus G. Melchers, Cornelius J. Konig, Peter A. Heslin, Filip Lievens

Research Collection Lee Kong Chian School Of Business

Personality assessments are often distorted during personnel selection, resulting in a common "ideal-employee factor" (IEF) underlying ratings of theoretically unrelated constructs. However, this seems not to affect the personality measures' criterion-related validity. The current study attempts to explain this set of findings by combining the literature on response distortion with the ones on cognitive schemata and on candidates' ability to identify criteria (ATIC). During a simulated selection process, 149 participants filled out Big Five personality measures and participated in several high- and low-fidelity work simulations to estimate their managerial performance. Structural equation modeling showed that the IEF presents an indicator …