Chapter 5: Exploring The Mind Of The Interviewer: Findings From Research With Interviewers To Improve The Survey Process Appendix 5, Robin Kaplan, Erica Yu

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 5A Interview Protocol

Appendix 5B Vignettes

Chapter 9: Why Do Interviewers Vary In Achieving Interview Privacy And Does Privacy Matter? Appendix 9, Zeina N. Mneimneh, Julie A. De Jong, Yasmin A. Altwaijri

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 9A

Table A9A.1 Description and univariate distribution of measures used in models

Appendix 9B

Table A9B.1 Random Intercept Two Level Logistic Regression Model Predicting Third-party Presence during the Interview

Chapter 3: General Interviewing Techniques: Developing Evidence-Based Practices For Standardized Interviewing Appendix 3, Nora Cate Schaeffer, Jennifer Dykema, Steve M. Coombs, Rob K. Schultz, Lisa Holland, Margaret L. Hudson

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Table A3A.1 Summary of Basic Techniques of Standardized Interviewing (adapted from Fowler and Mangione 1990, pp. 35-53)

Table A3A.2 Basic Question Forms (Response Formats)

Chapter 4: How To Conduct Effective Interviewer Training: A Meta-Analysis And Systematic Review Appendix 4, Jessica Daikeler, Michael Bosnjak

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 4A Appendix Table A4A.1: Overview of the literature on interviewer tasks addressed in interviewer training experiments

Appendix 4B The meta-analytical process

Appendix 4C Random effects model and meta regression summary statistics

Appendix 4D List of missing studies in the paper

References

Chapter 8: Examining The Utility Of Interviewer Observations On The Survey Response Process. Appendix 8, Brady T. West, Ting Yan, Frauke Kreuter, Michael Josten, Heather Schroeder

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 8A Additional Evaluation of Derived NSFG Classes

Figure A8A.1

Figure A8A.2

Table A8A.1

Table A8A.2

Table A8A.3

Appendix 8B Additional Details on ESS Items

Chapter 7: Statistical Identification Of Fraudulent Interviews In Surveys: Improving Interviewer Controls Appendix 7, Silvia Schwanhäuser, Joseph W. Sakshaug, Yuliya Kosyakova, Frauke Kreuter

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Table A7A.1 Number of identical response patterns

Chapter 6: Behavior Change Techniques For Reducing Interviewer Contributions To Total Survey Error Appendix 6, Brad Edwards, Hanyu Sun, Ryan Hubbard

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 6A Medical Expenditure Panel Survey key statistics for prescribed medicines for those 65 and older in the U.S.

Appendix 6B CAPI screenshots

Appendix 6C Westat’s CARI code screenshots

Appendix 6D Coding scheme

Appendix 6E Flowchart of CARI rapid feedback process

Appendix 6F Multilevel multinomial logistic regression models

Appendix 6G Multilevel multinomial logistic regression models with interviewer experience

Appendix 6H Alert frequency

Chapter 12: Differences In Interaction Quantity And Conversational Flow In Capi And Cati Interviews. Appendix 12, Yfke Ongena, Marieke Haan

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 12A Differences in Length of Interviewer-Respondent Interactions in CAPI and CATI Interviews

Table A12A.1 Response Rrates (AAPOR RR1)

Table A12A.2 Descriptive Statistics for Questions and Respondent Characteristics

Table A12A.3 Means and Correlations Trimmed number of Turns and Events in CAPI and CATI by Question Characteristics

Table A12A.4 Means and Correlations Trimmed number of Turns and Events by Question Characteristics

Chapter 11: Virtual Interviewers, Social Identities, And Survey Measurement Error. Appendix 11, Frederick Conrad, Michael Schober, Daniel Nielsen, Heidi Reichert

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Table A11A.1 Proportion of race-related items for which race of interviewer effects are observed across eleven published studies.

Table A11A.2 Virtual interviewer assignments and linked example videos, N=1,735

Table A11A.3. Debriefing questionnaire about respondents’ experience administered textually after the primary date were collected

Table A11A.4 Respondent – Virtual interviewer gender and race assignments, and match conditions N=1,735

Table A11A.5 Respondent characteristics, N=1,735

Table A11A.6 Questionnaire administered to respondents.

Table A11A.7 Respondent gender and race choices, N=1,735

Chapter 10: Unintended Interviewer Bias In A Community-Based Participatory Research Randomized Control Trial Among American Indian Youth Appendix 10, Patrick Habecker, Jerreed Ivanich

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 10A Analysis of Internalizing and Externalizing Subscales

Table A10A.1: Linear Mixed-Effects Models Predicting Subscales of Internalizing Behavior

Table A10A.2: Linear Mixed-Effects Models Predicting Subscales of Externalizing Behavior

Chapter 18: Response Times As An Indicator Of Data Quality: Associations With Question, Interviewer, And Respondent Characteristics In A Health Survey Of Diverse Respondents. Appendix 18, Dana Garbarski, Jennifer Dykema, Nora Cate Schaeffer, Dorothy Farrar Edwards

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 18A Description of individual question characteristics and hypotheses for their relationship with RTs

Appendix 18B Description of established tools for evaluating questions and hypotheses for their relationship with RTs

Appendix 18C Sample Description

Table 18.C1. Number of completed interviews by respondents’ race/ethnicity and sample

Appendix 18D Additional Tables

Appendix 18E References

Chapter 14: Explaining Interviewer Effects On Survey Unit Nonresponse: A Cross-Survey Analysis. Appendix 14, Daniela Ackermann-Piek, Julie M. Korbmacher, Ulrich Krieger

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 14A Questionnaire for the Interviewer Survey

Table A14A.1 Original German Version and English Translation of the Questionnaire for the GIP 2012, PIAAC, GIP 2014, and SHARE Interviewer Surveys

Appendix 14A: Descriptive Statistics: The Interviewers of the GIP 2012, PIAAC, GIP 2014, and SHARE

Table A14B.1 Descriptive Statistics for Interviewer’s Socio-Demographic Characteristics, Work Experience, and Working Hours per Week, Separate by Survey

Table A14B.1 Cronbach´s Alpha for Indicators on Interviewer Characteristics, all Surveys Combined and Separate by Survey

Table A14B.2 Interviewer Survey: Factor Matrix for Items About Reasons for Working as an Interviewer, Including Variances

Table A14B.3 Interviewer Survey: Factor …

Chapter 13: Interacting With Interviewers In Text And Voice Interviews On Smartphones. Appendix 13, Michael F. Schober, Frederick G. Conrad, Christopher Antoun, Alison W. Bowers, Andrew L. Hupp, H. Yanna Yan

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix A: Example human text and voice interchange that includes clarification.

Appendix B: Coding Manual

Appendix A13C.1 (Data) attached below

Chapter 17: Exploring The Antecedents And Consequences Of Interviewer Reading Speed (Irs) At The Question Level. Appendix 17, Allyson L. Holbrook, Timothy P. Johnson, Evgenia Kapousouz, Young Ik Cho

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Figure A17.A.1: Manipulation of Question Characteristics (Example Questions Shown)

Figure A17.A.2: Response Latency Validity Options Provided to Interviewers after Each Question where Response Latencies were Measured

Figure A17.A.3: Interviewer Behavior Codes Used to Identify Question Latency Problems

Appendix 17.B: Measurement of Response and Question Latencies Table A17.B.1: Validity of Response Latency Measurement
Table A17.B.2: Validity of Question Latency Measurement
References
Appendix 17.C: Questions in CAPI Survey for which Response Latencies were Measured

Chapter 16: Investigating The Use Of Nurse Paradata In Understanding Nonresponse To Biological Data Collection. Appendix 16, Fiona Pashazadeh, Alexandru Cernat, Joseph W. Sakshaug

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 16A: Descriptive statistics for available call records

Figure A16A.1 Histograms of wave 2 nurse visit call length by call status with outlier of 691 minutes removed from the ‘any interviewing done’ category

Figure A16A.2 Histograms of wave 3 nurse visit call length by call status with outlier of 464 minutes removed from the ‘any interviewing done’ category

Figure A16A.3 Bar chart of total number of calls per household at UKHLS wave 2 nurse visit

Figure A16A.4 Bar chart of total number of calls per household at UKHLS wave 3 nurse visit

Figure A16A.5 Histogram of total nurse visit time …

Chapter 22: A Comparison Of Different Approaches To Examining Whether Interviewer Effects Tend To Vary Across Different Subgroups Of Respondents. Appendix 22a, Geert Loosveldt, Celine Wuyts

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Table A22A.1 Substantive Questions about Climate Change and Energy (Module D) and Welfare Attitudes (Module E) Included in the Analysis

Syntax

Chapter 21: Modeling Interviewer Effects In The National Health Interview Survey. Appendix 21, James Dahlhamer, Aaron Maitland, Benjamin Zablotsky, Carla Zelaya

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Supplemental Table A21.1 Questions, Question Characteristics, and Intra-Interviewer Correlations (IIC)

Table A21.2 Descriptive Statistics for Respondent and Case Characteristics Included in Multi-Level Models

Table A21.3 Descriptive Statistics for County Measures Included in Multi-Level Models

Table A21.4 Descriptive Statistics for Interviewer Characteristics Included in Multi-Level Models

Table A21.5 Mock Dataset Structure Depicting Questions, Interviewer Groups, and IICs

Chapter 20: What Do Interviewers Learn? Changes In Interview Length And Interviewer Behaviors Over The Field Period. Appendix 20, Kristen M. Olson, Jolene Smyth

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 20A Full Model Coefficients and Standard Errors Predicting Count of Questions with Individual Interviewer Behaviors, Two-level Multilevel Poisson Models with Number of Questions Asked as Exposure Variable, WLT1 and WLT2

Analytic strategyTable A20A.1 Coefficients and Standard Errors from Multilevel Poisson Regression Models Predicting Number of Questions with Exact Question Reading with Total Number of Questions Asked to Each Respondent as an Exposure Variable, WLT1 and WLT2

Table A20A.2 Coefficients and Standard Errors from Multilevel Poisson Regression Models Predicting Number of Questions with Nondirective Probes with Total Number of Questions Asked to Each Respondent as an Exposure Variable, WLT1 and …

Chapter 19: Accuracy And Utility Of Using Paradata To Detect Question-Reading Deviations. Appendix 19, Jennifer Kelley

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Appendix 19A

Minor Deviations

Question as it Appears in Questionnaire

Examples of Deviations*

General Interviewer Techniques: Developing Evidence-Based Practices For Standardized Interviewing, Nora Cate Schaeffer, Jennifer Dykema, Steve M. Coombs, Rob K. Schultz, Lisa Holland, Margaret Hudson

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

The practices of standardized interviewing developed at many research sites over many years. The version of standardization that Fowler and Mangione codified in Standardized Survey Interviewing has provided researchers a core resource to use in training and supervising standardized interviewers. In recent decades, however, the accumulation of recordings and transcripts of interviews makes it possible to re-visit the practices of standardization to describe both how respondents actually answer survey questions and how interviewers actually respond.

To update General Interviewer Training (GIT), we brought observations of interaction during interviews together with research about conversational practices from conversation analysis, psychology, and other …

Antecedents And Consequences Of Interviewer Pace: Assessing Interviewer Speaking Pace At The Question Level, Allyson L. Holbrook, Timothy P. Johnson, Evgenia Kapousouz, Young Ik Cho

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

The pace at which interviewers read survey questions may vary considerably across interviewers (e.g., Cannell, Miller, & Oksenberg, 1981) and as a function of interviewer experience (Olson and Petchev, 2007). The pace at which interviews are conducted can influence respondent perceptions of the importance of interaction (Fowler, 1966). Interviewer training typically includes instructions to read questions slowly and clearly to respondents is based on the assumption that doing so maximizes data quality (e.g., Fowler and Mangione, 1990). In this research, we examine possible causes and consequences of interviewer pace using data from in person surveys conducted with respondents from four …

Exploring The Impact Of Interviewer Perceptions And Interviewer-Respondent Interactions On The Survey Of Income And Program Participation: Analysis Of Cari Recordings, Erica Yu, Rodney L. Terry, Alina Kline, Holly Fee, Robin Kaplan

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Interviewers play a significant role in telephone and face-to-face interviews, including gaining respondent cooperation and administering survey questions. Increasingly, interviewers’ perceptions of the respondent and interview experience, such as cooperativeness and interest, are also being used to assess measurement error and make adjustments to data (West, 2013; Kirchner et al., 2017). Although interviewer perceptions are typically recorded at the end of the interview, interviewers are likely to begin forming perceptions about the household and respondent based on their first contact attempt (and continue developing them during the interview). We hypothesize that interview context factors, such as interviewer perceptions of the …

Response Times As An Indicator Of Data Quality: Associations With Interviewer, Respondent, And Question Characteristics In A Health Survey Of Diverse Respondents, Dana Garbarski, Jennifer Dykema, Nora Cate Schaeffer, Dorothy Farrar Edwards

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Survey research remains one of the most important ways that researchers learn about key features of populations. Data obtained in the survey interview are a collaborative achievement accomplished through the interplay of the interviewer, respondent, and survey instrument, yet our field is still in the process of comprehensively documenting and examining whether, when, and how characteristics of interviewers, respondents, and questions combine to influence the quality of the data obtained.

Researchers tend to consider longer response times as indicators of potential problems as they indicate longer processing or interaction from the respondent, the interviewer (where applicable), or both. Previous work …

Race-Of-Virtual-Interviewer Effects, Frederick Conrad, Michael Schober, Daniel Nielsen, Heidi Reichert

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

In developing self-administered interviewing systems that go beyond text, survey designers are faced with choices about how to represent the interviewing agent. In speech-dialog systems like ACASI and IVR, designers must decide if the voice that presents the spoken questions is unambiguously male or female, whether the pronunciation is regionally marked, etc. Any visual representation of an interviewer (e.g., a photograph, a video) requires designers to choose features that visually convey demographic features like race, gender, age, etc. Here we investigate whether the representation of animated virtual interviewers (VIs) affects responses in the same way that analogous attributes of human …

How To Conduct Effective Interviewer Training: A Meta-Analysis, Jessica Daikeler

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Interviewer training can improve the performance of interviewers and thus also the quality of survey data. However, the question of how effective interviewer training is for improving data quality and more importantly, which determinates drive its success, remain unanswered. This research uses meta-analytical methods to evaluate both the improvements in data quality due to interviewer training and the effectivity of training modules with respect interviewer performance. We consider various aspects of data quality, namely unit nonresponse, item nonresponse, probing behavior, administration, reading, and recording. Based on more than sixty experimental studies, we find that comprehensive interviewer training improves unit- and …

What Do Interviewers Learn? Changes In Interview Length And Interviewer Behaviors Over The Field Period, Kristen Olson, Jolene Smyth

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Interviewers systematically speed up over the field period of a survey as they conduct interviews (Olson and Peytchev 2007; Olson and Bilgen 2011; Kirchner and Olson 2017). Competing hypotheses for this increase in speed is that interviewers learn from previous interviews, changing their behaviors accordingly, or that they change behaviors in response to who the respondent is, including both respondent’s fixed characteristics and their response propensity. Previous work (e.g., Kirchner and Olson 2017) has failed to completely explain this learning effect, even after accounting for a wide range of measures of each of these hypotheses. However, prior work has not …

Interviewer Variation In Third Party Presence During Face-To-Face Interviews, Zeina N. Mneimneh, Julie De Jong, Jennifer Kelley

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

The presence of a third person in face-to-face interviews constitutes an important contextual factor that affects the interviewee's responses to culturally sensitive questions (Aquilino, 1997; Casterline and Chidambaram, 1984; Mneimneh et al., 2015; Pollner and Adams, 1994). Interviewers play an essential role in requesting, achieving, and reporting on the private setting of the interview. Our recent work has shown that the rate of interview privacy varies significantly across interviewers; while some interviewers report high rates of privacy among their interviews, others report low rates of privacy for the interviews they administered (Mneimneh et al., 2018). Yet, there is a lack …

The Cannell Legacy, Nancy A. Mathiowetz, Peter V. Miller

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Charles Cannell engineered the study of Interviewer-respondent interaction. He created conceptual frameworks for understanding the interviewing process and its impact on data quality. He invented methods for observing and recording interview interaction. He pioneered the use of randomized experiments in the survey context. He amalgamated insights from clinical and social psychology, sociology, group dynamics, as well as research on verbal and nonverbal communication to inform his work. This interdisciplinary approach has broadly influenced both interviewing research and practice. In this paper, we review Cannell’s many contributions to the field and his enduring legacy.

Scientific Network Of Experts: Interviewer Effects And Interviewer Training, Daniela Ackermann-Piek, Joe Sakshaug

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

Although the collection of survey data is undergoing a notable shift toward online and mixed-mode data collection methods (Baker et al., 2010; Groves, 2011), interviewers are still heavily involved in the majority of survey data collections that serve as a basis for important economic, educational, and public policy decisions. Research supports the notion that interviewer characteristics and task-specific skill levels significantly influence the resulting data quality (see, e.g. Ackermann-Piek, 2018; Billiet & Loosveldt, 1988; Dahlhamer, Cynamon, Gentleman, Piani, & Weiler, 2010; Durand, 2005; Fowler Jr., 1991; Hox & de Leeuw, 2002; Jäckle, Lynn, Sinibaldi, & Tipping, 2013; Sakshaug, Tutz, & …

Interacting With Interviewers In Voice And Text Interviews On Smartphones, Michael Schober, Frederick Conrad, Christopher Antoun, Alison W. Bowers, Andrew L. Hupp, H. Yanna Yan

Interviewer Workshop, 2019: Interviewers and Their Effects from a Total Survey Error Perspective

As people increasingly adopt SMS text messaging for communicating in their daily lives, texting becomes a potentially important way to interact with survey respondents, who may expect that they can communicate with survey researchers as they communicate with others. Thus far our evidence from analyses of 642 iPhone interviews suggests that text interviewing can lead to higher quality data (less satisficing, more disclosure) than voice interviews on the same device, whether the questions are asked by an interviewer or an automated system. Respondents also report high satisfaction with text interviews, with many reporting that text is more convenient because they …