Skip to main content
Springer Nature Link
Log in

Different Sample Sources, Different Results? A Comparison of Online Panel and Mail Survey Respondents

  • Conference paper
  • First Online:
Design, User Experience, and Usability: UX Research, Design, and Assessment (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13321))

Included in the following conference series:

  • 2601 Accesses

Abstract

This paper compares data and results from two different survey modes: a probability sampled postal survey and a nonprobability sampled online panel. Our main research objective was to explore if there are differences between the sample methods in terms of nonresponse, item response bias, and selectivity. Both the postal survey and online panel data consist of Finns aged 18–74. Altogether, 2470 respondents were included in the probability sample gathered randomly from the population register of Finland (sample size was 8000 with a response rate of 30.9%), and 1254 respondents were from an online panel organized by a market company. We collected the data in late 2017. The findings confirmed that an online panel can improve the representativeness by including more respondents from groups that are underrepresented within the traditional probability sample. However, we found that panel respondents were more likely to leave unanswered questions perceived as sensitive, which may be a sign of a measurement bias related to intrusiveness. Moreover, the results indicated selection differences between samples related to respondents’ media interests.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Due to the very small amount of Finnish speakers, the Åland-area was excluded from the study.

References

  1. Stoop, I., Billiet, J., Koch, A., Fitzgerald, R.: Improving Survey Response: Lessons Learned from the European Social Survey (2010)

    Google Scholar 

  2. Hays, R.D., Liu, H., Kapteyn, A.: Use of Internet panels to conduct surveys. Behav. Res. Methods 47, 685–690 (2015)

    Article  Google Scholar 

  3. Hox, J.J., De Leeuw, E.D., Zijlmans, E.A.O.: Measurement equivalence in mixed mode surveys. Front Psychol.6, 87 (2015)https://doi.org/10.3389/fpsyg.2015.00087

  4. Dillman, D.A., Smyth, J.D., Christian, L.M.: Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th edn. Wiley, New Jersey (2014)

    Google Scholar 

  5. Dillman, D.A., Hao, F., Millar, M.M.: The SAGE handbook of online research methods. In: Fielding, N.G., Lee, R.M., Blank, G. (eds.) The SAGE Handbook of Online Research Methods. SAGE, Los Angeles, pp. 220–240 (2017)

    Google Scholar 

  6. Atkeson, L.R., Adams, A.N.: Mixing survey modes and its implications. In: Atkeson, L.R., Alvarez, R.M. (eds.) The Oxford Handbook of Polling and Survey Methods, pp. 53–75. Oxford University Press, Oxford (2018)

    Google Scholar 

  7. Pohjanoksa-Mäntylä, M., Turunen, J.: Kyselytutkimus. In: Hämeen-Anttila, K., Katajavuori, N. (eds.) Yhteiskunnallinen lääketutkimus – ideasta näyttöön, 2nd edn., pp. 80–96. University of Helsinki, Helsinki (2021)

    Google Scholar 

  8. Rich, R.C., Brians, C.L., Manheim, J.B., Willnat, L.: Empirical Political Analysis: Quantitative and Qualitative Research Methods, 9th edn. Routledge, New York (2018)

    Book  Google Scholar 

  9. Grönlund, K., Strandberg, K.: Online panel research : representativeness and attrition in the finnish eopinion panel. In: Callegaro, M., Baker, R.P., Bethlehem, J., et al. (eds.) Online Panel Research : A Data Quality Perspective, pp. 86–103. John Wiley & Sons, Chichester (2014)

    Chapter  Google Scholar 

  10. Lehdonvirta, V., Oksanen, A., Räsänen, P., Blank, G.: Social media, web, and panel surveys: using non-probability samples in social and policy research. Policy Internet 13, 134–155 (2021). https://doi.org/10.1002/POI3.238

    Article  Google Scholar 

  11. Koivula, A., Sirppiniemi, R., Koiranen, I., Oksanen, J.: Workingpapers in Economic Sociology: Arkielämä ja osallistuminen -kyselyn tutkimusseloste. University of Turku, Department of Social Research, Turku (2017)

    Google Scholar 

  12. Van Loon, A.J.M., Tijhuis, M., Picavet, H.S.J., et al.: Survey non-response in the Netherlands: effects on prevalence estimates and associations. Ann. Epidemiol. 13, 105–110 (2003). https://doi.org/10.1016/S1047-2797(02)00257-0

    Article  Google Scholar 

  13. Antholz, B.: Cover letter reduces response rate. Bull Me´thodologie Sociol 137(1), 140–156 (2018)

    Google Scholar 

  14. Saari, H., Koivula, A., Sivonen, J., Räsänen, P.: Working papers in Economic Sociology : Suomi 2019 – kulutus ja elämäntapa. Tutkimusseloste ja koodikirja (2019). https://www.utupub.fi/handle/10024/148680

    Google Scholar 

  15. Rosentiel, T., Witt, E., Best, J.: How Different Are People Who Don’t Respond to Pollsters? | Pew Research Center. In: Pew Res. Cent. (2008). https://www.pewresearch.org/2008/04/21/how-different-are-people-who-dont-respond-to-pollsters/. Accessed 4 Feb 2022

  16. Ramo, D.E., Prochaska, J.J.: Broad reach and targeted recruitment using facebook for an online survey of young adult substance use. J. Med. Internet Res. 14, e1878(2012). https://doi.org/10.2196/JMIR.1878

  17. Retention of College Students with Type 1 Diabetes via Social Media: An Implementation Case Study. J. Diabetes Sci. Technol. 13, 445–456. https://doi.org/10.1177/1932296819839503

  18. Survey Research: Using Facebook and Instagram Advertisements and In-Person Intercept in LGBT Bars and Nightclubs to Recruit LGBT Young Adults. J. Med. Internet Res. 20(6), e197 (2018). https://www.jmir.org/2018/6/e197. https://doi.org/10.2196/JMIR.9461

  19. Blom, A.G., Gathmann, C., Krieger, U.: Setting up an online panel representative of the general population: the German internet panel. Field Methods 27, 391–408 (2015)

    Article  Google Scholar 

  20. Tourangeau, R., Yan, T.: Sensitive questions in surveys. Psychol. Bull 133, 859–883 (2007). https://doi.org/10.1037/0033-2909.133.5.859

    Article  Google Scholar 

  21. Ziegenfuss, J.Y., Easterday, C.A., Dinh, J.M., et al.: Impact of demographic survey questions on response rate and measurement: a randomized experiment. Surv. Pract. 14(1), 26126 (2021)

    Google Scholar 

  22. Pollien, A., Herzing, J.M.E., Antal, E.: Preparation of survey data : FORS Guide No. 13, Version 1.0. Swiss Centre of Expertise in the Social Sciences (FORS) (2020)

    Google Scholar 

  23. Gnambs, T., Kaspar, K.: Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behav. Res. Methods 47, 1237–1259 (2015). https://doi.org/10.3758/S13428-014-0533-4/FIGURES/3

    Article  Google Scholar 

  24. Naus, M.J., Philipp, L.M., Samsi, M.: From paper to pixels: a comparison of paper and computer formats in psychological assessment. Comput. Human Behav. 25, 1–7 (2009). https://doi.org/10.1016/J.CHB.2008.05.012

    Article  Google Scholar 

  25. Kays, K., Gathercoal, K., Buhrow, W.: Does survey format influence self-disclosure on sensitive question items? Comput. Human Behav. 28, 251–256 (2012). https://doi.org/10.1016/J.CHB.2011.09.007

    Article  Google Scholar 

  26. Huang, H.M.: Do print and Web surveys provide the same results? Comput. Human Behav. 22, 334–350 (2006). https://doi.org/10.1016/J.CHB.2004.09.012

    Article  Google Scholar 

  27. Heerwegh, D., Loosveldt, G.: Face-to-face versus web surveying in a high-internet-coverage population differences in response quality. Public Opin. Q 72, 836–846 (2008). https://doi.org/10.1093/POQ/NFN045

    Article  Google Scholar 

  28. de Leeuw, E.D., Hox, J.J., Boeve, A.: Handling do-not-know answers: exploring new approaches in online and mixed-mode surveys. Soc. Sci. Comput. Rev. 32, 116–132 (2016)

    Article  Google Scholar 

  29. Matthijsse, S.M., De Leeuw, E.D., Hox, J.J.: Internet panels, professional respondents, and data quality. Methodology 11, 81–88 (2015). https://doi.org/10.1027/1614-2241/a000094

    Article  Google Scholar 

  30. Goyder, J.: The Silent Minority: Non-respondents in Sample Surveys. Routledge, New York (2019)

    Book  Google Scholar 

  31. Zillmann, D., Schmitz, A., Skopek, J., Blossfeld, H.-P.: Survey topic and unit nonresponse: evidence from an online survey on mating. Qual. Quant. 48, 2069–2088 (2014)

    Article  Google Scholar 

  32. Keusch, F., Batinic, B., Mayerhofer, W.: Motives for joining nonprobability online panels and their association with survey participation behavior. In: Online Panel Research. John Wiley & Sons, Ltd., Chichester, UK, pp. 171–191 (2014)

    Google Scholar 

  33. Sivonen, J., Koivula, A., Saarinen, A., Keipi, T.: Working Papers in Economic Sociology : Research Report on the Finland in the Digital Age -Survey. University of Turku, Department of Social Research, Turku (2018)

    Google Scholar 

  34. Lee, C.K., Back, K.J., Williams, R.J., Ahn, S.S.: Comparison of telephone RDD and online panel survey modes on CPGI scores and co-morbidities. Int. Gambl. Stud. 15, 435–449 (2015). https://doi.org/10.1080/14459795.2015.1068353

    Article  Google Scholar 

  35. Hemsworth, L.M., Rice, M., Hemsworth, P.H., Coleman, G.J.: Telephone survey versus panel survey samples assessing knowledge, attitudes and behavior regarding animal welfare in the red meat industry in Australia. Front Psychol. 12, 1024 (2021). https://doi.org/10.3389/FPSYG.2021.581928/BIBTEX

    Article  Google Scholar 

  36. Walter, S.L., Seibert, S.E., Goering, D., O’Boyle, E.H.: A tale of two sample sources: do results from online panel data and conventional data converge? J. Bus Psychol. 34, 425–452 (2019). https://doi.org/10.1007/S10869-018-9552-Y/TABLES/6

    Article  Google Scholar 

  37. Eyal, P., David, R., Andrew, G., et al.: Data quality of platforms and panels for online behavioral research. Behav. Res. Methods (2021)

    Google Scholar 

  38. Bendau, A., Petzold, M.B., Pyrkosch, L., et al.: Associations between COVID-19 related media consumption and symptoms of anxiety, depression and COVID-19 related fear in the general population in Germany. Eur. Arch. Psychiatry Clin. Neurosci. 271, 283–291 (2021). https://doi.org/10.1007/S00406-020-01171-6

    Article  Google Scholar 

  39. Nekliudov, N.A., Blyuss, O., Cheung, K.Y., et al.: Excessive media consumption about COVID-19 is associated with increased state anxiety: outcomes of a large online survey in Russia. J. Med. Internet Res. 22, e20955 (2020). https://doi.org/10.2196/20955

  40. Nabi, R.L., Wolfers, L.N., Nathan, W., Qi, L.: Coping with COVID-19 stress: the role of media consumption in emotion- and problem-focused coping. Psychol. Pop Media (2022)

    Google Scholar 

  41. Callegaro, M., Baker, R.P., Bethlehem, J., et al.: Online panel research: history, concepts, applications and a look at the future. In: Callegaro, M., Baker, R.P., Bethlehem, J., et al. (eds.) Online Panel Research : A Data Quality Perspective, pp. 1–54. John Wiley & Sons, Chichester (2014)

    Chapter  Google Scholar 

  42. Karlson, K.B., Holm, A., Breen, R.: Comparing regression coefficients between same-sample nested models using logit and probit a new method. Sociol. Methodol. 42, 286–313 (2012)

    Article  Google Scholar 

  43. Statistics Finland. Statistics Finland’s PX-Web databases: Statfin (2018). http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin/?rxid=99142dcd-2c78-437d-8172-6a68fbadccfa. Accessed 15 May 2018

  44. Bradley, V.C., Kuriwaki, S., Isakov, M., et al.: Unrepresentative big surveys significantly overestimated US vaccine uptake. Nat 6007890(600), 695–700 (2021). https://doi.org/10.1038/s41586-021-04198-4

    Article  Google Scholar 

  45. Coppock, A., Leeper, T.J., Mullinix, K.J.: Generalizability of heterogeneous treatment effect estimates across samples. Proc. Natl. Acad. Sci. 115, 12441–12446 (2018). https://doi.org/10.1073/PNAS.1808083115

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Social Research, University of Turku, Turku, Finland

    Aki Koivula & Jukka Sivonen

Authors
  1. Aki Koivula
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jukka Sivonen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jukka Sivonen .

Editor information

Editors and Affiliations

  1. Southern University of Science and Technology – SUSTech, Shenzhen, China

    Marcelo M. Soares

  2. World Usability Day and Bubble Mountain Consulting, Newton Center, MA, USA

    Elizabeth Rosenzweig

  3. Aaron Marcus and Associates, Berkeley, CA, USA

    Aaron Marcus

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Koivula, A., Sivonen, J. (2022). Different Sample Sources, Different Results? A Comparison of Online Panel and Mail Survey Respondents. In: Soares, M.M., Rosenzweig, E., Marcus, A. (eds) Design, User Experience, and Usability: UX Research, Design, and Assessment. HCII 2022. Lecture Notes in Computer Science, vol 13321. Springer, Cham. https://doi.org/10.1007/978-3-031-05897-4_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05897-4_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05896-7

  • Online ISBN: 978-3-031-05897-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics