Proven and New Survey Techniques to Obtain Sensitive Information from Consumers

Researchers often wish to complement observed choice data with survey information to gain a deeper understanding of the underlying attitudes, motivations, and mechanisms that drive consumption behavior. Surveys are fundamental to obtain information that is unavailable through objectively recorded official statistics or about behaviors that are hard to measure. However, consumers are increasingly reluctant to share information about themselves in surveys: frequent privacy violations and breaches have made them more circumspect. Furthermore, topics that were once nonsensitive have now become stigmatized or politicized. For instance, consumers might claim to make sustainable or “green” consumption choices due to social desirability (White, Habib, and Hardisty 2019). Due to these trends, obtaining accurate information about sensitive topics has become increasingly challenging to analysts.

This report summarizes our framework of strategies that are available to analysts to enhance truthful and accurate survey responses on sensitive topics. We present a decision tree to aid survey design for sensitive topics. The decision tree addresses presurvey administration, question design to improve truth-telling (survey planning), and statistical corrections after data collection (postsurvey adjustments).

We complement the decision tree with eight survey techniques to reduce social desirability bias, summarized in the table below. We rate each of these techniques on the following criteria that influence truthful reporting:

• Privacy protection: The degree of confidentiality guaranteed to survey participants, where “very high” indicates that the survey designer cannot infer the individual answer.
• Efficiency: Some techniques introduce noise in the response to hide participants’ answers. As a consequence, accuracy will be lower and statistics of interest will be imperfectly estimated, with large standard errors.
• Affective costs: Some techniques might require survey participants to (seemingly) incriminate themselves which may be emotionally stressful.
• Cognitive costs: Instructions of some techniques may be difficult for survey participants to understand.
• Design complexity (behavioral costs): Some techniques may be complex to implement, requiring much effort for appropriate implementation.

High affective and cognitive costs may result in systematic bias and/or survey attrition: uncomfortable or confused participants may drop out, or provide untruthful or random answers. The tree requires analysts to consider the alphabetically labeled questions as well as their implications for the survey design and analysis; in addition, it introduces several techniques to reduce social desirability bias. We elaborate on each numbered step in the tree.

1. Use direct questions if the topic is not sensitive. Before introducing unnecessary complexity in the survey design, topic sensitivity should be assessed (Tourangeau and Yan 2007). Perhaps what the analysts consider to be a sensitive topic is not as sensitive for survey participants (for instance, soft drug use among students). If in doubt, a pretest can be used to ask a sample of eligible participants to evaluate the sensitivity of certain topics or questions. Alternatively, the survey can ask participants to evaluate topic sensitivity ex post and assess whether the reported sensitivity is correlated with the responses to sensitive questions. If evidence of topic sensitivity is found, this can be reduced by building trust with the participant.
   
   
2. Stress anonymity and allow for opt-out. Building trust with the survey participant before the actual survey is essential to obtain accurate responses about potentially sensitive topics. Trust can be enhanced by using easily understandable confidentiality and ethics guarantees and by clearly stating the purpose of the survey and the intended data use (Singer, Thurn, and Miller 1995; Tourangeau et al. 1997; Tourangeau and Yan 2007). These guarantees should emphasize participants’ legal right to privacy and state that participants may opt out from participating at any time during the survey. Further, these guarantees should stress the importance of participants’ answers to better understand behavioral phenomena. Stantcheva (2023) provides more guidelines and suggestions. Data quality can also be enhanced by asking participants to commit to provide thoughtful answers (Geisen 2022), similarly to “honesty pledges.” Finally, trust can be built by allowing survey participants to bypass potentially sensitive questions. Skipping could be introduced with a “prefer not to say” answer option, or by allowing survey participants to skip a sensitive section entirely (Pieters and De Jong 2024).
   
   
3. Reduce sensitivity of direct questions. If indirect questions are not possible or optimal to obtain the information of interest, social desirability bias in direct questions can be reduced, for instance by using more neutral and less emotional words, or by extending the time frame of interest to a longer period. Open-ended questions allow participants more flexibility in terms of how much to disclose.
   
   
4. Use indirect evaluation questions: Consumers are sometimes not consciously aware of their true behaviors or preferences—for instance, when reporting on snacking frequency, stereotyping or gender issues. At the same time, they project their own behaviors, beliefs, and evaluations onto others (Fisher 1993). Indirect evaluation questions exploit their inability to gauge their true preferences and evaluation biases to obtain truthful responses to sensitive questions. For instance, “social circle” questions ask about the behavior of neighbors or friends. Social circle questions improved survey polling for the U.S. presidential election in 2016 (Galesic et al. 2018). Related to this approach, the Bayesian truth serum asks survey participants to predict their own behavior as well as its prevalence in the group. Survey participants are compensated according to the accuracy of their prediction, creating incentives for accurate reporting (Weaver and Prelec 2013).
   
   A related indirect question approach is the endorsement experiment, where survey participants are asked to endorse a person or organization. These are randomly associated (or not) with a controversial policy or topic (e.g., abortion in Rosenfeld, Imai, and Shapiro [2016]). Endorsement of a certain organization or person may thus reveal hidden information about participants’ thoughts on certain sensitive topics.
   
   
5. Mask responses for sample-level information (simple). If aggregate data at the sample level are sufficient, responses can be simply masked, so that the individual response becomes unobservable. For instance, list experiments ask survey participants to answer how many questions from a set of questions they endorse (De Jong and Pieters 2019). Participants are randomly assigned to two conditions, each with the same baseline k questions, but with an extra-sensitive question in one of the conditions. The analyst can then simply infer from the difference in average response between conditions which proportion of the sample endorses the sensitive question. Randomized response techniques hide answers by using randomization devices, such as (electronic) dice or coins, whose outcome is masked to the survey designer (De Jong, Pieters, and Fox 2010; Kwan, So, and Tam 2010). For instance, they might randomly assign survey participants to a sensitive or a nonsensitive question, making unclear which question the participant is actually answering. The Crosswise Model presents instead survey participants with a pair of items, one sensitive and one nonsensitive, after which they are asked whether the answers to the two statements are the same or different (Yu, Tian, and Tang 2008). These techniques have lower efficiency, as they introduce additional noise, and thus are best suited when survey topics are strongly socially desirable, such as tobacco smoking among pregnant mothers.
   
   
6. Mask responses for individual-level information (advanced). When individual-level data on sensitive topics is needed, and the previous steps of our framework have been taken, analysts could make use of new survey techniques to obtain individual-level data from response masking. These new techniques use statistical techniques to infer probabilistic individual-level data from masked survey responses, while still protecting participant privacy (De Jong and Pieters 2019; Gregori, De Jong, and Pieters 2024; Kuha and Jackson 2014). The key idea behind these techniques is that the probability that a participant provides specific answers to the baseline questions (in case of list surveys, but similar ideas hold for other techniques) can be estimated from other information in the survey, and from that the participant’s response to the sensitive questions can be inferred probabilistically.
   
   
7. Use direct questions with social desirability responding corrections. The previous set of techniques aim to prevent social desirability bias. Early literature on social desirability responding (SDR) introduced partial correlation techniques to correct biased responses to sensitive, direct questions. First, participants’ inclination to provide socially acceptable responses, that is, their propensity to SDR, is assessed using so-called lie scales. These lie scales contain items such as “I have not always been honest with myself” or “I sometimes tell lies if I have to” (Hart et al. 2015). SDR propensity can thus be quantified and can then be included as a covariate when predicting the responses to sensitive questions. This should correct for variation in the outcome variable due to SDR propensity. Importantly, SDR correction of potentially biased responses to direct questions about sensitive topics is commonly not very effective and the extent of validity improvement often unknown (Lanz, Thielmann, and Gerpott 2022). Steenkamp et al. (2010) provide a procedure for using SDR scales to improve validity of marketing construct.

Summary

Enhancing perceived and actual privacy of survey participants is a rich, active research area. In this short article, we have outlined a number of strategies that can be utilized to build trust with survey participants and ultimately elicit truthful responses to sensitive questions. These techniques should be combined with ex post data processing to ensure that sensitive individual information cannot be deanonymized when making the data available (see, e.g., Gupta, Moutafis, and Schneider 2022). Furthermore, additional issues need to be considered when designing surveys; in particular, respondents’ fatigue, lack of engagement, or boredom might reduce the external validity of survey results (see Li et al. 2022). We hope this short introduction provides some accessible guidance towards obtaining sensitive information from consumers.

References

De Jong, Martijn G., and Rik Pieters (2019), “Assessing Sensitive Consumer Behavior Using the Item Count Response Technique,” Journal of Marketing Research, 56 (3), 345–60. https://doi.org/10.1177/0022243718821312

De Jong, Martijn G., Rik Pieters, and Jean-Paul Fox (2010), “Reducing Social Desirability Bias Through Item Randomized Response: An Application to Measure Underreported Desires,” Journal of Marketing Research, 47(1), 14–27. https://doi.org/10.1509/jmkr.47.1.14

Fisher, Robert J. (1993). “Social Desirability Bias and the Validity of Indirect Questioning,” Journal of Consumer Research, 20 (2), 303–15. https://doi.org/10.1086/209351

Galesic, M., W. Bruine de Bruin, M. Dumas, A. Kapteyn, J.E. Darling, and E. Meijer (2018), “Asking About Social Circles Improves Election Predictions,” Nature Human Behaviour, 2 (3), 187–93. https://www.nature.com/articles/s41562-018-0302-y

Geisen, Emily (2022), “Improve Data Quality by Using Commitment Request Instead of Attention Checks,” Qualtrics XM Blog (August 4), https://www.qualtrics.com/blog/attention-checks-and-data-quality/.

Gregori, Marco, Martijn G. De Jong, and Rik Pieters (2024), “Response Aggregation to Obtain Truthful Answers to Sensitive Questions: Estimating the Prevalence of Illegal Purchases of Prescription Drugs,” Journal of Marketing Research, https://doi.org/10.1177/00222437231205252

Gupta, Sachin, Panos Moutafis, and Matthew J. Schneider (2022), “The Marketer at the Privacy Table,” Impact at JMR (March 17), https://www.ama.org/2022/03/17/the-marketer-at-the-privacy-table/.

Hart, Claire M., Timothy D. Ritchie, Erica G. Hepper, and Jochen E. Gebauer (2015), “The Balanced Inventory of Desirable Responding Short Form (BIDR-16),” Sage Open, 5(4), https://doi.org/10.1177/2158244015621113.

Kuha, Jouni and Jonathan Jackson (2014), “The Item Count Method for Sensitive Survey Questions: Modelling Criminal Behaviour,” Journal of the Royal Statistical Society Series C: Applied Statistics, 63 (2), 321–41. https://www.jstor.org/stable/24771848

Kwan, Samuel S., Mike K. So, and Kar Yan Tam (2010), “Research note—Applying the Randomized Response Technique to Elicit Truthful Responses to Sensitive Questions in IS Research: The Case of Software Piracy Behavior,” Information Systems Research, 21 (4), 941–59. https://doi.org/10.1287/isre.1090.0271

Yu, Jun-Wu, Guo-Liang Tian, and Man-Lai Tang (2008), “Two New Models for Survey Sampling with Sensitive Characteristic: Design and Analysis,” Metrika, 67 (2008), 251–63. https://link.springer.com/article/10.1007/s00184-007-0131-x

Lanz, Lukas, Isabel Thielmann, and Fabiola H. Gerpott (2022), “Are Social Desirability Scales Desirable? A Meta‐Analytic Test of the Validity of Social Desirability Scales in the Context of Prosocial Behavior,” Journal of Personality, 90 (2), 203–21. https://doi.org/10.1111/jopy.12662

Li, Ye, Antonia Krefeld-Schwalb, Daniel G. Wall, Eric J. Johnson, Olivier Toubia, and Daniel M. Bartels (2022), “The More You Ask, the Less You Get: When Additional Questions Hurt External Validity,” Journal of Marketing Research, 59 (5), 963–82. https://doi.org/10.1177/00222437211073581

Meckel, Katherine and Bradley Shapiro (2021), “Depression and Shopping Behavior,” Working Paper No. w29462, National Bureau of Economic Research, https://www.nber.org/papers/w29462.

Rosenfeld, Bryn, Kosuke Imai, and Jacob N. Shapiro (2016), “An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions,” American Journal of Political Science, 60 (3), 783–802. https://doi.org/10.1111/ajps.12205

Pieters, Rik and Martijn G. De Jong (2024), “The Commodification of Intimacy: Desires for Escort Services,” working paper.

Singer, Eleanor, Dawn R. Von Thurn, and Esther R. Miller (1995), “Confidentiality Assurances and Response: A Quantitative Review of the Experimental Literature,” Public Opinion Quarterly, 59 (1), 66–77. https://doi.org/10.1086/269458

Stantcheva, Stefanie (2023), “How to Run Surveys: A Guide to Creating Your Own Identifying Variation and Revealing the Invisible,” Annual Review of Economics, 15, 205–34. https://doi.org/10.1146/annurev-economics-091622-010157

Steenkamp, Jan-Benedict E.M., Martijn G. De Jong, and Hans Baumgartner (2010), “Socially Desirable Response Tendencies in Survey Research,” Journal of Marketing Research, 47 (2), 199–214. https://doi.org/10.1509/jmkr.47.2.199

Tourangeau, R., J.B. Jobe, W.F. Pratt, and K. Rasinski (1997), “Design and Results of the Women’s Health Study,” NIDA Research Monograph, 167, 344–65. https://pubmed.ncbi.nlm.nih.gov/9243569/

Tourangeau, Roger, and Ting Yan (2007), “Sensitive Questions in Surveys,” Psychological Bulletin, 133 (5), 859–83. https://doi.org/10.1037/0033-2909.133.5.859

Weaver, Ray and Drazen Prelec (2013), “Creating Truth-Telling Incentives with the Bayesian Truth Serum,” Journal of Marketing Research, 50 (3), 289–302. https://doi.org/10.1509/jmr.09.0039

Wetzel, Eunike, Susanne Frick, and Anna Brown (2021), “Does Multidimensional Forced-Choice Prevent Faking? Comparing The Susceptibility of the Multidimensional Forced-Choice Format and the Rating Scale Format to Faking,” Psychological Assessment, 33 (2), 156–70. https://doi.org/10.1037/pas0000971

White, Katherine, Rishad Habib, and David J. Hardisty (2019), “How to SHIFT Consumer Behaviors to Be More Sustainable: A Literature Review and Guiding Framework,” Journal of Marketing, 83 (3), 22–49. https://doi.org/10.1177/0022242919825649

Decision Tree Step	Technique	Privacy Protection	Efficiency	Affective Costs	Cognitive Costs	Design Complexity	Example Formulation/Reference
1. Direct questions	i. Direct questions	Very low	Very high	Very high	Very low	Very easy
2. Opt-out	ii. Skip questions	Low	High	Moderate	Very low	Very easy	The next questions can be sensitive. It is no problem if you do not want to answer these questions. Please indicate here: “proceed,” “skip these questions.”  See Pieters and De Jong (2024).
4. Indirect evaluation questions	iii. Social circle questions	Moderate	High	Moderate	Low	Very easy	Indicate how many of your social contacts engage in the following behavior/have the following attitudes.
	iv. Bayesian truth serum	Moderate	High	Moderate	Low	Difficult	How often do you <sensitive behavior>? Predict the percentage of respondents who choose each option. See Weaver and Prelec (2013).
	v. Endorsement experiment	Very high	Very low	Very low	Low	Difficult	Do you have a favorable opinion of the following <organization/person>? See Rosenfeld, Imai and Shapiro (2016).
5/6. Response masking	vi. List experiment	High	Low	Low	Low	Difficult	How many of the next <number> statements are true?
	vii. Randomized response	Very high	Low	Moderate	Moderate	Moderate	Toss a coin. Answer “yes” if the coin toss comes up heads. Otherwise, please answer the following question.
	viii. Crosswise Model/Paired Response Technique	Very high	Low	Very low	Low	Moderate	Are your answer to the next two statements the same or different? See Gregori, De Jong and Pieters (2023).

Notes: The table is based on Rosenfeld, Imai, and Shapiro (2016) and Gregori, De Jong, and Pieters (2023). Ranking of techniques is by the current authors based on overall superiority (most to least).

Go to the Journal of Marketing Research