Chapter 19: Accuracy And Utility Of Using Paradata To Detect Question-Reading Deviations. Appendix 19, Jennifer Kelley

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 19A

Minor Deviations

Question as it Appears in Questionnaire

Examples of Deviations*

Are Self-Description Scales Better Than Agree/Disagree Scales?, Jerry Timbrook, Jolene Smyth, Kristen M. Olson

Department of Sociology: Faculty Publications

Questions using agree/disagree (A/D) scales are ubiquitous in survey research because they save time and space on questionnaires through display in grids, but they have also been criticized for being prone to acquiescent reports. Alternatively, questions using self-description (SD) scales (asking respondents how well a statement describes them from Completely to Not at All) can also be presented in grids or with a common question stem, and by omitting the word agree, SD scales may reduce acquiescence. However, no research has examined how response patterns may differ across A/D and SD scales. In this article, we compare survey …

Chapter 5: Exploring The Mind Of The Interviewer: Findings From Research With Interviewers To Improve The Survey Process Appendix 5, Robin Kaplan, Erica Yu

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 5A Interview Protocol

Appendix 5B Vignettes

Chapter 4: How To Conduct Effective Interviewer Training: A Meta-Analysis And Systematic Review Appendix 4, Jessica Daikeler, Michael Bosnjak

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 4A Appendix Table A4A.1: Overview of the literature on interviewer tasks addressed in interviewer training experiments

Appendix 4B The meta-analytical process

Appendix 4C Random effects model and meta regression summary statistics

Appendix 4D List of missing studies in the paper

References

Chapter 9: Why Do Interviewers Vary In Achieving Interview Privacy And Does Privacy Matter? Appendix 9, Zeina N. Mneimneh, Julie A. De Jong, Yasmin A. Altwaijri

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 9A

Table A9A.1 Description and univariate distribution of measures used in models

Appendix 9B

Table A9B.1 Random Intercept Two Level Logistic Regression Model Predicting Third-party Presence during the Interview

Chapter 8: Examining The Utility Of Interviewer Observations On The Survey Response Process. Appendix 8, Brady T. West, Ting Yan, Frauke Kreuter, Michael Josten, Heather Schroeder

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 8A Additional Evaluation of Derived NSFG Classes

Figure A8A.1

Figure A8A.2

Table A8A.1

Table A8A.2

Table A8A.3

Appendix 8B Additional Details on ESS Items

Chapter 6: Behavior Change Techniques For Reducing Interviewer Contributions To Total Survey Error Appendix 6, Brad Edwards, Hanyu Sun, Ryan Hubbard

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 6A Medical Expenditure Panel Survey key statistics for prescribed medicines for those 65 and older in the U.S.

Appendix 6B CAPI screenshots

Appendix 6C Westat’s CARI code screenshots

Appendix 6D Coding scheme

Appendix 6E Flowchart of CARI rapid feedback process

Appendix 6F Multilevel multinomial logistic regression models

Appendix 6G Multilevel multinomial logistic regression models with interviewer experience

Appendix 6H Alert frequency

Chapter 10: Unintended Interviewer Bias In A Community-Based Participatory Research Randomized Control Trial Among American Indian Youth Appendix 10, Patrick Habecker, Jerreed Ivanich

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 10A Analysis of Internalizing and Externalizing Subscales

Table A10A.1: Linear Mixed-Effects Models Predicting Subscales of Internalizing Behavior

Table A10A.2: Linear Mixed-Effects Models Predicting Subscales of Externalizing Behavior

Chapter 14: Explaining Interviewer Effects On Survey Unit Nonresponse: A Cross-Survey Analysis. Appendix 14, Daniela Ackermann-Piek, Julie M. Korbmacher, Ulrich Krieger

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 14A Questionnaire for the Interviewer Survey

Table A14A.1 Original German Version and English Translation of the Questionnaire for the GIP 2012, PIAAC, GIP 2014, and SHARE Interviewer Surveys

Appendix 14A: Descriptive Statistics: The Interviewers of the GIP 2012, PIAAC, GIP 2014, and SHARE

Table A14B.1 Descriptive Statistics for Interviewer’s Socio-Demographic Characteristics, Work Experience, and Working Hours per Week, Separate by Survey

Table A14B.1 Cronbach´s Alpha for Indicators on Interviewer Characteristics, all Surveys Combined and Separate by Survey

Table A14B.2 Interviewer Survey: Factor Matrix for Items About Reasons for Working as an Interviewer, Including Variances

Table A14B.3 Interviewer Survey: Factor …

Chapter 18: Response Times As An Indicator Of Data Quality: Associations With Question, Interviewer, And Respondent Characteristics In A Health Survey Of Diverse Respondents. Appendix 18, Dana Garbarski, Jennifer Dykema, Nora Cate Schaeffer, Dorothy Farrar Edwards

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 18A Description of individual question characteristics and hypotheses for their relationship with RTs

Appendix 18B Description of established tools for evaluating questions and hypotheses for their relationship with RTs

Appendix 18C Sample Description

Table 18.C1. Number of completed interviews by respondents’ race/ethnicity and sample

Appendix 18D Additional Tables

Appendix 18E References

Chapter 17: Exploring The Antecedents And Consequences Of Interviewer Reading Speed (Irs) At The Question Level. Appendix 17, Allyson L. Holbrook, Timothy P. Johnson, Evgenia Kapousouz, Young Ik Cho

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Figure A17.A.1: Manipulation of Question Characteristics (Example Questions Shown)

Figure A17.A.2: Response Latency Validity Options Provided to Interviewers after Each Question where Response Latencies were Measured

Figure A17.A.3: Interviewer Behavior Codes Used to Identify Question Latency Problems

Appendix 17.B: Measurement of Response and Question Latencies Table A17.B.1: Validity of Response Latency Measurement
Table A17.B.2: Validity of Question Latency Measurement
References
Appendix 17.C: Questions in CAPI Survey for which Response Latencies were Measured

Chapter 16: Investigating The Use Of Nurse Paradata In Understanding Nonresponse To Biological Data Collection. Appendix 16, Fiona Pashazadeh, Alexandru Cernat, Joseph W. Sakshaug

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 16A: Descriptive statistics for available call records

Figure A16A.1 Histograms of wave 2 nurse visit call length by call status with outlier of 691 minutes removed from the ‘any interviewing done’ category

Figure A16A.2 Histograms of wave 3 nurse visit call length by call status with outlier of 464 minutes removed from the ‘any interviewing done’ category

Figure A16A.3 Bar chart of total number of calls per household at UKHLS wave 2 nurse visit

Figure A16A.4 Bar chart of total number of calls per household at UKHLS wave 3 nurse visit

Figure A16A.5 Histogram of total nurse visit time …

Chapter 22: A Comparison Of Different Approaches To Examining Whether Interviewer Effects Tend To Vary Across Different Subgroups Of Respondents. Appendix 22a, Geert Loosveldt, Celine Wuyts

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Table A22A.1 Substantive Questions about Climate Change and Energy (Module D) and Welfare Attitudes (Module E) Included in the Analysis

Syntax

Chapter 21: Modeling Interviewer Effects In The National Health Interview Survey. Appendix 21, James Dahlhamer, Aaron Maitland, Benjamin Zablotsky, Carla Zelaya

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Supplemental Table A21.1 Questions, Question Characteristics, and Intra-Interviewer Correlations (IIC)

Table A21.2 Descriptive Statistics for Respondent and Case Characteristics Included in Multi-Level Models

Table A21.3 Descriptive Statistics for County Measures Included in Multi-Level Models

Table A21.4 Descriptive Statistics for Interviewer Characteristics Included in Multi-Level Models

Table A21.5 Mock Dataset Structure Depicting Questions, Interviewer Groups, and IICs

Chapter 20: What Do Interviewers Learn? Changes In Interview Length And Interviewer Behaviors Over The Field Period. Appendix 20, Kristen M. Olson, Jolene Smyth

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 20A Full Model Coefficients and Standard Errors Predicting Count of Questions with Individual Interviewer Behaviors, Two-level Multilevel Poisson Models with Number of Questions Asked as Exposure Variable, WLT1 and WLT2

Analytic strategyTable A20A.1 Coefficients and Standard Errors from Multilevel Poisson Regression Models Predicting Number of Questions with Exact Question Reading with Total Number of Questions Asked to Each Respondent as an Exposure Variable, WLT1 and WLT2

Table A20A.2 Coefficients and Standard Errors from Multilevel Poisson Regression Models Predicting Number of Questions with Nondirective Probes with Total Number of Questions Asked to Each Respondent as an Exposure Variable, WLT1 and …

The Effect Of Question Characteristics On Question Reading Behaviors In Telephone Surveys, Kristen M. Olson, Jolene Smyth, Antje Kirchner

Department of Sociology: Faculty Publications

Asking questions fluently, exactly as worded, and at a reasonable pace is a fundamental part of a survey interviewer’s role. Doing so allows the question to be asked as intended by the researcher and may decrease the risk of measurement error and contribute to rapport. Despite the central importance placed on reading questions exactly as worded, interviewers commonly misread questions, and it is not always clear why. Thus, understanding the risk of measurement error requires understanding how different interviewers, respondents, and question features may trigger question reading problems. In this article, we evaluate the effects of question features on question …

A Comparison Of Fully Labeled And Top-Labeled Grid Question Formats, Jolene Smyth, Kristen M. Olson

Department of Sociology: Faculty Publications

The grid question format is common in mail and web surveys. In this format, a single question stem introduces a set of items, which are listed in rows of a table underneath the question stem. The table’s columns contain the response options, usually only listed at the top, with answer spaces arrayed below and aligned with the items (Dillman et al. 2014).This format is efficient for respondents; they do not have to read the full question stem and full set of response options for every item in the grid. Likewise, it is space efficient for the survey researcher, which reduces …

Unpacking The Black Box Of Survey Costs, Kristen M. Olson

Department of Sociology: Faculty Publications

Survey costs are a critically important input to and constraint on the quality of data collected from surveys. Much about survey costs is unknown, leading to lack of understanding of the drivers of survey costs, the relationship between survey costs and survey errors, and difficulty in justifying the importance of survey data versus other available administrative or organic data. This commentary outlines a recently developed typology for survey costs, illustrates this typology using methodological articles that report on costs in pharmacy surveys, and provides recommendations for research on the relationship between fixed and variable costs as a major area for …

Nebraska Annual Social Indicators Survey (Nasis) 2020 Winter Methodology Report, Bureau Of Sociological Research

Nebraska Annual Social Indicators Survey (NASIS)

No abstract provided.

Cafe Erc Nasis Survey Questions 2019 Winter Report, Julia Mcquillan

Nebraska Annual Social Indicators Survey (NASIS)

Using NSF ERC Planning Grant funds ($7,500), we added X number of survey items to the 2019 Winter Nebraska Annual Social Indicators Survey (NASIS) conducted by the Bureau of Sociological Research (BOSR). In 2019 NASIS was conducted by mail. BOSR mailed surveys to a random sample of Nebraska households using Address Based Sampling. The CAFE ERC planning team added questions to NASIS along with other researchers at the University of Nebraska who shared the cost of the “core” questions (e.g. level of education, age, race/ethnicity, depression, religion, political orientation, quality-of-life topics, etc). Overall NASIS is a cost-effective way to collect …

Nebraska Annual Social Indicators Survey (Nasis) 2019 Methodology Report, Bureau Of Sociological Research

Nebraska Annual Social Indicators Survey (NASIS)

Introduction 3

Mode Selection 3

Design and Item Selection 3

Sampling Design 4

Experimental Design Treatment 4

Data Collection Process 5

Response Rate 5

Data-Entry Training, Supervision, and Quality Control 5

Processing of Completed Surveys 6

Data Cleaning 6

NASIS Sample Weights 6

Design Effects 7

Questions 7

Estimate of Sampling Error 10

Appendices 11

Appendix A: Cover Letters and NASIS 2019 FAQ 11

First Mailing – Version 1 (UNL Logo) 11

FAQ – Version 1 (UNL Logo) 13

FAQ – Version 2 (New UN Logo) 14

Second Mailing – Version 1 (UNL Logo) 15

Second Mailing – Version 2 …

Analysis Of Social Unrest Events Using Spatio-Temporal Data Clustering And Agent-Based Modelling, Sudeep Basnet

School of Computing: Dissertations, Theses, and Student Research

Social unrest such as appeals, protests, conflicts, fights and mass violence can result from a wide ranging of diverse factors making the analysis of causal relationships challenging, with high complexity and uncertainty. Unrest events can result in significant changes in a society ranging from new policies and regulations to regime change. Widespread unrest often arises through a process of feedback and cascading of a collection of past events over time, in regions that are close to each other. Understanding the dynamics of these social events and extrapolating their future growth will enable analysts to detect or forecast major societal events. …

Nebraska Annual Social Indicators Survey (Nasis) 2019 Winter Methodology Report, Bureau Of Sociological Research

Nebraska Annual Social Indicators Survey (NASIS)

Bureau of Sociological Research

2019 Winter NASIS Methodology Report

Contents

Introduction 3

Mode Selection 3

Design and Item Selection 3

Sampling Design 4

Data Collection Process 4

Response Rate 5

Data-Entry Training, Supervision, and Quality Control 5

Processing of Completed Surveys 5

Data Cleaning 5

NASIS Sample Weights 6

Design Effects 6

Questions 6

Estimate of Sampling Error 9

Appendices 10

Appendix A: Cover Letters and 2019 Winter NASIS FAQ 10

First Mailing – Cover Letter 10

First Mailing – FAQ 11

Last Mailing – Cover Letter 12

Appendix B: Survey Instrument (Printed in black & white only) 14

Appendix …

General Interviewer Techniques: Developing Evidence-Based Practices For Standardized Interviewing, Nora Cate Schaeffer, Jennifer Dykema, Steve M. Coombs, Rob K. Schultz, Lisa Holland, Margaret Hudson

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

The practices of standardized interviewing developed at many research sites over many years. The version of standardization that Fowler and Mangione codified in Standardized Survey Interviewing has provided researchers a core resource to use in training and supervising standardized interviewers. In recent decades, however, the accumulation of recordings and transcripts of interviews makes it possible to re-visit the practices of standardization to describe both how respondents actually answer survey questions and how interviewers actually respond.

To update General Interviewer Training (GIT), we brought observations of interaction during interviews together with research about conversational practices from conversation analysis, psychology, and other …

Antecedents And Consequences Of Interviewer Pace: Assessing Interviewer Speaking Pace At The Question Level, Allyson L. Holbrook, Timothy P. Johnson, Evgenia Kapousouz, Young Ik Cho

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

The pace at which interviewers read survey questions may vary considerably across interviewers (e.g., Cannell, Miller, & Oksenberg, 1981) and as a function of interviewer experience (Olson and Petchev, 2007). The pace at which interviews are conducted can influence respondent perceptions of the importance of interaction (Fowler, 1966). Interviewer training typically includes instructions to read questions slowly and clearly to respondents is based on the assumption that doing so maximizes data quality (e.g., Fowler and Mangione, 1990). In this research, we examine possible causes and consequences of interviewer pace using data from in person surveys conducted with respondents from four …

Race-Of-Virtual-Interviewer Effects, Frederick Conrad, Michael Schober, Daniel Nielsen, Heidi Reichert

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

In developing self-administered interviewing systems that go beyond text, survey designers are faced with choices about how to represent the interviewing agent. In speech-dialog systems like ACASI and IVR, designers must decide if the voice that presents the spoken questions is unambiguously male or female, whether the pronunciation is regionally marked, etc. Any visual representation of an interviewer (e.g., a photograph, a video) requires designers to choose features that visually convey demographic features like race, gender, age, etc. Here we investigate whether the representation of animated virtual interviewers (VIs) affects responses in the same way that analogous attributes of human …

Interviewer Variation In Third Party Presence During Face-To-Face Interviews, Zeina N. Mneimneh, Julie De Jong, Jennifer Kelley

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

The presence of a third person in face-to-face interviews constitutes an important contextual factor that affects the interviewee's responses to culturally sensitive questions (Aquilino, 1997; Casterline and Chidambaram, 1984; Mneimneh et al., 2015; Pollner and Adams, 1994). Interviewers play an essential role in requesting, achieving, and reporting on the private setting of the interview. Our recent work has shown that the rate of interview privacy varies significantly across interviewers; while some interviewers report high rates of privacy among their interviews, others report low rates of privacy for the interviews they administered (Mneimneh et al., 2018). Yet, there is a lack …

The Cannell Legacy, Nancy A. Mathiowetz, Peter V. Miller

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Charles Cannell engineered the study of Interviewer-respondent interaction. He created conceptual frameworks for understanding the interviewing process and its impact on data quality. He invented methods for observing and recording interview interaction. He pioneered the use of randomized experiments in the survey context. He amalgamated insights from clinical and social psychology, sociology, group dynamics, as well as research on verbal and nonverbal communication to inform his work. This interdisciplinary approach has broadly influenced both interviewing research and practice. In this paper, we review Cannell’s many contributions to the field and his enduring legacy.

The Accuracy And Utility Of Using Paradata To Detect Interviewer Question-Reading Deviations, Jennifer Kelley

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Deviations from reading survey questions exactly as worded may change the validity of the questions, thus increasing measurement error. Hence, organizations train their interviewers to read questions verbatim. To ensure interviewers are reading questions verbatim, organizations rely on interview recordings. However, this takes a significant amount of resources. Therefore, some organizations are using paradata generated by the survey software, specifically timestamps, to try to detect when interviewers’ deviate from reading the question verbatim.

To monitor interviewers’ question reading behavior using timestamps, some organizations estimate the expected question administration time to establish a minimum and maximum question administration time thresholds (QATT). …

Interacting With Interviewers In Voice And Text Interviews On Smartphones, Michael Schober, Frederick Conrad, Christopher Antoun, Alison W. Bowers, Andrew L. Hupp, H. Yanna Yan

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

As people increasingly adopt SMS text messaging for communicating in their daily lives, texting becomes a potentially important way to interact with survey respondents, who may expect that they can communicate with survey researchers as they communicate with others. Thus far our evidence from analyses of 642 iPhone interviews suggests that text interviewing can lead to higher quality data (less satisficing, more disclosure) than voice interviews on the same device, whether the questions are asked by an interviewer or an automated system. Respondents also report high satisfaction with text interviews, with many reporting that text is more convenient because they …