Social and Behavioral Sciences Commons^™

Crowdsourcing Hypothesis Tests: Making Transparent How Design Choices Shape Research Results, Justin F. Landy, Miaolei Jia, Isabel L. Ding, Domenico Viganola, Warren Tierney, Andree Hartanto

Research Collection School of Social Sciences

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to …

A Practical Guide For Using Electronic Surveys In Aviation Research: Best Practices Explained, Stephen Rice, Scott R. Winter

International Journal of Aviation, Aeronautics, and Aerospace

The purpose of this article is to provide aviation researchers who are interested in using Internet-based surveys with a justification and guide, along with best practices. A brief review of research methods, research design, and research methodologies precedes information on the types of research questions that can best be answered using survey research. The interaction of survey instruments and research design is discussed, such as using a survey instrument in an experimental design. Advantages and disadvantages of paper and electronic distribution are presented to help guide researchers into the best approach given their proposed goals. Guidance is also offered on …

Crowdsourcing Job Satisfaction Data: Examining The Construct Validity Of Glassdoor.Com Ratings, Richard N. Landers, Robert C. Brusso, Elena M. Auer

Personnel Assessment and Decisions

Researchers, practitioners, and job seekers now routinely use crowdsourced data about organizations for both decision-making and research purposes. Despite the popularity of such websites, empirical evidence regarding their validity is generally absent. In this study, we tackled this problem by combining two curated datasets: (a) the results of the 2017 Federal Employee Viewpoint Survey (FEVS), which contains facet-level job satisfaction ratings from 407,789 US federal employees, and which we aggregated to the agency level, and (b) current overall and facet ratings of job satisfaction of the federal agencies contained within FEVS from Glassdoor.com as scraped from the Glassdoor application programming …

Introduction To The Special Issue On Advanced Technologies In Assessment: A Science-Practice Concern, Tara S. Behrend, Richard N. Landers

Personnel Assessment and Decisions

No abstract provided.

The Psychological Science Accelerator: Advancing Psychology Through A Distributed Collaborative Network, Hannah Moshontz, Lorne Campbell, Charles R. Ebersole, Hans Ijzerman, Heather L. Urry, Patrick S. Forscher, Jon E. Grahe, Randy J. Mccarthy, Erica D. Musser, Jan Antfolk, Christopher M. Castille, Thomas Rhys Evans, Susann Fiedler, Jessica Kay Flake, Diego A. Forero, Steve M. Janssen, Justin Robert Keene, John Protzko, Balazs Aczel, Sara Álvarez Solas, Daniel Ansari, Dana Awlia, Ernest Baskin, Carlota Batres, Martha Lucia Borras-Guevara, Cameron Brick, Priyanka Chandel, Armand Chatard, William J. Chopik, David Clarance, Nicholas A. Coles, Katherine S. Corker

Brain and Mind Institute Researchers' Publications

Concerns have been growing about the veracity of psychological research. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions, or attempt to replicate prior …

The Psychological Science Accelerator: Advancing Psychology Through A Distributed Collaborative Network, Hannah Moshontz, Lorne Campbell, Charles R. Ebersole, Hans Ijzerman, Heather L. Urry, Patrick S. Forscher, Jon E. Grahe, Randy J. Mccarthy, Erica D. Musser, Jan Antfolk

Psychological Science Faculty Publications and Presentations

Concerns have been growing about the veracity of psychological research. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions, or attempt to replicate prior …

Crowdsourcing Consciousness: You Think, Therefore I Am, Justin M. Campbell

Undergraduate Honors Capstone Projects

The challenge to understand consciousness is a centuries-old interdisciplinary research program. The search entails fundamental questions about our nature - the desire to understand who we are has been around for nearly as long as experience itself. It is also one of the most important questions we can ask; meaning itself is predicated on having some sort of conscious experiencer for whom something can matter. Given the magnitude and intractability of explaining the paradox of how consciousness can be at once the most obvious thing in the universe, and also the most inaccessible, the endeavor is a tremendous undertaking. Until …

Unique Prediction Of Cannabis Use Severity And Behaviors By Delay Discounting And Behavioral Economic Demand, Justin C. Strickland, Joshua A. Lile, William W. Stoops

Psychology Faculty Publications

Few studies have simultaneously evaluated delay discounting and behavioral economic demand to determine their unique contribution to drug use. A recent study in cannabis users found that monetary delay discounting uniquely predicted cannabis dependence symptoms, whereas cannabis demand uniquely predicted use frequency. This study sought to replicate and extend this research by evaluating delay discounting and behavioral economic demand measures for multiple commodities and including a use quantity measure. Amazon.com’s Mechanical Turk was used to sample individuals reporting recent cannabis use (n = 64) and controls (n = 72). Participants completed measures of monetary delay discounting as well as alcohol …

Many Hands Make Light Work: Crowdsourced Ratings Of Medical Student Osce Performance, Mark Grichanik

USF Tampa Graduate Theses and Dissertations

Clinical skills are often measured using objective structured clinical examinations (OSCEs) in healthcare professions education programs. As with assessment centers, it is challenging to provide learners with effective feedback due to burdensome human capital demands. The aim of this dissertation was to evaluate the viability of using a crowdsourced system to gather OSCE ratings and feedback. Aggregating evaluations of student performance from a crowd of patient proxies has the potential to mitigate biases associated with single-rater evaluations, allow the patient a voice as the consumer of physician behavior, improve reliability, reduce costs, improve feedback latency, and help learners develop a …

The Reliability Of Crowdsourcing: Latent Trait Modeling With Mechanical Turk, Matt Baucum, Steven Rouse Dr., Cindy Miller-Perrin, Elizabeth Mancuso Dr.

Seaver College Research And Scholarly Achievement Symposium

Mechanical Turk, an online crowdsourcing platform, has recently received increased attention in the social sciences as studies continue to suggest its viability as a source for reliable experimental data. Given the ease with which large samples can be quickly and inexpensively gathered, it is worth examining whether Mechanical Turk can provide accurate experimental data for methodologies requiring such large samples. One such methodology is Item Response Theory, a psychometric paradigm that defines test items by a mathematical relationship between a respondent’s ability and the probability of item endorsement. To test whether Mechanical Turk can serve as a reliable source of …

The Impact Of Culture On Creativity: How Cultural Tightness And Cultural Distance Affect Global Innovation Crowdsourcing Work, Roy Y. J. Chua, Yannig Roth, Jean-François Lemoine

Roy CHUA

This paper advances a new theoretical model to understand the effect of culture on creativity in a global context. We theorize that creativity engagement and success depend on the cultural tightness—the extent to which a country is characterized by strong social norms and low tolerance for deviant behaviors—of both an innovator’s country and the audience’s country, as well as the cultural distance between these two countries. Using field data from a global online crowdsourcing platform that organizes creative contests for consumer-product brands, supplemented by interviews with marketing experts, we found that individuals from tight cultures are less likely than counterparts …

Power To The People: A Comprehensive Look At Crowdsourcing Initiatives In Cultural Institutions, Danielle Pace

Seton Hall University Dissertations and Theses (ETDs)

Today, crowdsourcing has become an integrative approach to completing projects using the help of the general populous. These projects aid museum staff by processing large quantities of data, which otherwise could not be completed due to time and/or staff restraints. Through crowdsourcing, cultural institutions have the ability to outsource these tasks to volunteers, who can complete them at much faster rates. Although staff members are needed to validate and supervise these projects, crowdsourcing remains a useful tool in increasing public interactions and project efficiency.

This thesis presents a thorough outline of what crowdsourcing is, how it is being utilized, and …