Alamir Novin
Eric Meyers
ABSTRACT
Currently, there is widespread media coverage about the problems with 'fake news' that appears in social media, but the effects of biased information that appears in search engine results is also increasing. The authors argue that the search engine results page (SERP) exposes three important types of bias: source bias, algorithmic bias, and cognitive bias. To explore the relationship between these three types of bias, we conducted a mixed methods study with sixty participants (plus fourteen in a pilot to make a total of seventy-four participants). Within a library setting, participants were provided with mock search engine pages that presented order-controlled sources on a science controversy. Participants were then asked to rank the sources' usefulness and then summarize the controversy. We found that participants ranked the usefulness of sources depending on its presentation within a SERP. In turn, this also influenced how the participants summarized the topic. We attribute the differences in the participants' writings to the cognitive biases that affect a user's judgment when selecting sources on a SERP. We identify four main cognitive biases that a SERP can evoke in students: Priming, Anchoring, Framing, and the Availability Heuristic. While policing information quality is a quixotic task, changes can be made to both SERPs and a user's decision-making when selecting sources. As bias emerges both on the system side and the user side of search, we suggest a two-fold solution is required to address these challenges.
- N. J. Belkin, M. Cole, and J. Liu, "A model for evaluation of interactive information retrieval," presented at the Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, 2009, pp. 7--8.Google Scholar
- R. Epstein and R. E. Robertson, "The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections," Proc. Natl. Acad. Sci., p. 201419828, Aug. 2015.Google Scholar
- E. Hargittai, "Digital Na(t)ives" Variation in Internet Skills and Uses among Members of the "Net Generation,"? Sociol. Inq., vol. 80, no. 1, pp. 92--113, Feb. 2010.Google ScholarCross Ref
- E. M. Meyers, "Access denied: how students resolve information needs when an ideal document is restricted," in Proceedings of the 2012 iConference, 2012, pp. 629--631. Google ScholarDigital Library
- "National Science Board (2014) Science and Engineering Indicators." National Science Foundation, Arlington, VA., 2014.Google Scholar
- Q. Guo, R. W. White, S. T. Dumais, J. Wang, and B. Anderson, "Predicting query performance using query, result, and user interaction features," presented at the Adaptivity, Personalization and Fusion of Heterogeneous Information, 2010, pp. 198--201. Google ScholarDigital Library
- N. Höchstötter and D. Lewandowski, "What users see? Structures in search engine results pages," Inf. Sci., vol. 179, no. 12, pp. 1796--1812, 2009. Google ScholarDigital Library
- D. Kelly, "Methods for evaluating interactive information retrieval systems with users," Found. Trends Inf. Retr., vol. 3, no. 1--2, pp. 1--224, 2009. Google ScholarDigital Library
- A. Shiri and L. Zvyagintseva, "Dynamic Query Suggestion in Web Search Engines: A Comparative Examination," presented at the Proceedings of the Annual Conference of CAIS/Actes du congrès annuel de l-ACSI, 2014.Google Scholar
- E. R. Tufte, Beautiful evidence. Cheshire, Conn: Graphics Press, 2006. Google ScholarDigital Library
- R. Cellan-Jones, "Six searches that show the power of Google," Apr. 2016.Google Scholar
- S. Gerhart, "Do Web search engines suppress controversy",? First Monday, vol. 9, no. 1, 2004.Google Scholar
- D. C. Mutz, Population-based survey experiments. Princeton University Press, 2011.Google Scholar
- M. Schudson, "The 'Lippmann-Dewey Debate' and the Invention of Walter Lippmann as an Anti-Democrat 1985--1996," Int. J. Commun., vol. 2, p. 12, 2008.Google Scholar
- J. W. Carey, Communication as Culture: Essays on Media and Society. Psychology Press, 1989.Google Scholar
- J. Dewey and M. L. Rogers, The public and its problems: An essay in political inquiry. Penn State Press, 2012.Google Scholar
- W. Lippmann, Public opinion. Transaction Publishers, 1946.Google Scholar
- E. Amend and D. Barney, "Getting It Right: Canadian Conservatives and the 'War on Science'.," Can. J. Commun., vol. 41, no. 1, 2016.Google ScholarCross Ref
- S. Dunwoody, "Science journalism," Handb. Public Commun. Sci. Technol., pp. 15--26, 2008.Google Scholar
- E. Segev and A. J. Sharon, "Temporal patterns of scientific information-seeking on Google and Wikipedia," Public Underst. Sci., p. 963662516648565, 2016.Google Scholar
- B. Latour, Science in action: How to follow scientists and engineers through society. Harvard university press, 1987.Google Scholar
- L. Daston and P. Galison, "The Image of Objectivity," Representations, no. 40, pp. 81--128, Oct. 1992.Google ScholarCross Ref
- T. Bubela et al., "Science communication reconsidered," Nat. Biotechnol., vol. 27, no. 6, pp. 514--518, 2009.Google ScholarCross Ref
- L. D. Carsten and D. L. Illman, "Perceptions of accuracy in science writing," Prof. Commun. IEEE Trans. On, vol. 45, no. 3, pp. 153--156, 2002.Google ScholarCross Ref
- S. Dunwoody and M. Ryan, "Scientific barriers to the popularization of science in the mass media," J. Commun., vol. 35, no. 1, pp. 26--42, 1985.Google ScholarCross Ref
- McComas and Simone, "Media Coverage of Conflict Science Communication," Vol 24 No, vol. 24, no. 4, Jun. 2003.Google Scholar
- D. Nelkin, Selling Science - How the Press Covers Science and Technology. New York: W.H. Freeman and Company, 1987.Google Scholar
- D. A. Scheufele, "Science communication as political communication," Proc. Natl. Acad. Sci., vol. 111, no. Supplement_4, pp. 13585--13592, Sep. 2014.Google ScholarCross Ref
- G. C. Bowker and S. L. Star, Sorting things out classification and its consequences. Cambridge, Mass.: MIT Press, 1999. Google ScholarDigital Library
- E. Morozov, The net delusion: the dark side of internet freedom, 1. ed. New York, NY: Public Affairs, 2011. Google ScholarDigital Library
- B. J. Fogg, Persuasive Technology: Using Computers to Change what We Think and Do. Morgan Kaufmann, 2003. Google ScholarDigital Library
- B. Friedman and H. Nissenbaum, "Bias in computer systems," ACM Trans. Inf. Syst. TOIS, vol. 14, no. 3, pp. 330--347, 1996. Google ScholarDigital Library
- D. Kahneman and A. Tversky, "Subjective probability: A judgment of representativeness," Cognit. Psychol., vol. 3, no. 3, pp. 430--454, 1972.Google ScholarCross Ref
- D. Kahneman and A. Tversky, "On the reality of cognitive illusions.," 1996.Google ScholarCross Ref
- D. Kahneman, Thinking, fast and slow. 2013.Google Scholar
- S. Lewandowsky, U. K. H. Ecker, C. M. Seifert, N. Schwarz, and J. Cook, "Misinformation and Its Correction: Continued Influence and Successful Debiasing," Psychol. Sci. Public Interest, vol. 13, no. 3, pp. 106--131, Dec. 2012.Google ScholarCross Ref
- P. C. Wason and D. Shapiro, "Natural and contrived experience in a reasoning problem," Q. J. Exp. Psychol., vol. 23, no. 1, pp. 63--71, Feb. 1971.Google ScholarCross Ref
- C. Behnert and D. Lewandowski, "Ranking Search Results in Library Information Systems -- Considering Ranking Approaches Adapted From Web Search Engines," J. Acad. Librariansh., vol. 41, no. 6, pp. 725--735, Nov. 2015.Google ScholarCross Ref
- L. Lorigo et al., "Eye tracking and online search: Lessons learned and challenges ahead," J. Am. Soc. Inf. Sci. Technol., vol. 59, no. 7, pp. 1041--1052, 2008. Google ScholarDigital Library
- B. Pan, H. Hembrooke, T. Joachims, L. Lorigo, G. Gay, and L. Granka, "In google we trust: Users' decisions on rank, position, and relevance," J. Comput. Commun., vol. 12, no. 3, pp. 801--823, 2007.Google Scholar
- T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay, "Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search," ACM Trans. Inf. Syst. TOIS, vol. 25, no. 2, p. 7, 2007. Google ScholarDigital Library
- R. Fidel, Human information interaction: an ecological approach to information behavior. MIT Press, 2012. Google ScholarCross Ref
- D. O. Case, Looking for Information: A Survey of Research on Information Seeking, Needs and Behavior. Emerald Group Publishing, 2012.Google Scholar
- G. B. Chapman and E. J. Johnson, "Incorporating the irrelevant: Anchors in judgments of belief and value," Heuristics Biases Psychol. Intuitive Judgm., pp. 120--138, 2002.Google ScholarCross Ref
- J. S. B. T. Evans and D. E. Over, Rationality and Reasoning. Psychology Press, 2013.Google ScholarCross Ref
- R. Cellan-Jones, "The Force of Google," BBC Radio 4, May-2016.Google Scholar
- R. B. Zajonc, "Mere exposure: A gateway to the subliminal," Curr. Dir. Psychol. Sci., vol. 10, no. 6, pp. 224--228, 2001.Google ScholarCross Ref
- J. A. Bargh, P. M. Gollwitzer, A. Lee-Chai, K. Barndollar, and R. Trötschel, "The automated will: nonconscious activation and pursuit of behavioral goals.," J. Pers. Soc. Psychol., vol. 81, no. 6, p. 1014, 2001.Google ScholarCross Ref
- C. R. Mynatt, M. E. Doherty, and R. D. Tweney, "Confirmation bias in a simulated research environment: An experimental study of scientific inference," Q. J. Exp. Psychol., vol. 29, no. 1, pp. 85--95, Feb. 1977.Google ScholarCross Ref
- C. A. Chinn, "Factors that Influence," presented at the Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society: June 18 to 21, 1993, Institute of Cognitive Science, University of Colorado-Boulder, 1993, vol. 15, p. 318.Google Scholar
- L. Salmerón, Y. Kammerer, and P. García-Carrión, "Searching the Web for conflicting topics: Page and user factors," Comput. Hum. Behav., vol. 29, no. 6, pp. 2161--2171, Nov. 2013. Google ScholarDigital Library
- E. Agichtein, E. Brill, and S. Dumais, "Improving web search ranking by incorporating user behavior information," presented at the Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006, pp. 19--26. Google ScholarDigital Library
- J. Liu and N. J. Belkin, "Personalizing information retrieval for multi-session tasks: Examining the roles of task stage, task type, and topic knowledge on the interpretation of dwell time as an indicator of document usefulness," J. Assoc. Inf. Sci. Technol., vol. 66, no. 1, pp. 58--81, 2015.Google ScholarDigital Library
- N. J. Belkin, M. Cole, and J. Liu, "A model for evaluation of interactive information retrieval," presented at the Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, 2009, pp. 7--8.Google Scholar
- D. Case, "Sixty years of measuring the use of information and its sources: from consultation to application," Libr. Digit. Age LIDA Proc., vol. 13, 2014.Google Scholar
- D. O. Case and L. G. O'Connor, "What's the use? Measuring the frequency of studies of information outcomes: Journal of the Association for Information Science and Technology," J. Assoc. Inf. Sci. Technol., vol. 67, no. 3, pp. 649--661, Mar. 2016. Google ScholarDigital Library
- B. Pan, H. Hembrooke, T. Joachims, L. Lorigo, G. Gay, and L. Granka, "In Google We Trust: Users' Decisions on Rank, Position, and Relevance," J. Comput.-Mediat. Commun., vol. 12, no. 3, pp. 801--823, Apr. 2007.Google ScholarCross Ref
- Google, The Evolution of Search. 2011.Google Scholar
- A. Tversky and D. Kahneman, "Judgment under uncertainty: Heuristics and biases," science, vol. 185, no. 4157, pp. 1124--1131, 1974.Google Scholar
- M. Hilbert, "Toward a synthesis of cognitive biases: How noisy information processing can bias human decision making.," Psychol. Bull., vol. 138, no. 2, pp. 211--237, 2012.Google ScholarCross Ref
- E. Yong, "Psychologists strike a blow for reproducibility," Nature, Nov. 2013.Google Scholar
- C. Ware, Information visualization: perception for design, Third edition. Waltham, MA: Morgan Kaufmann, 2013. Google ScholarDigital Library
- D. Kahneman, A. Treisman, and B. J. Gibbs, "The reviewing of object files: Object-specific integration of information," Cognit. Psychol., vol. 24, no. 2, pp. 175--219, Apr. 1992.Google ScholarCross Ref
- N. Höchstötter and D. Lewandowski, "What users see -- Structures in search engine results pages," Inf. Sci., vol. 179, no. 12, pp. 1796--1812, May 2009. Google ScholarDigital Library
- M. R. Laurent and T. J. Vickers, "Seeking health information online: does Wikipedia matter",? J. Am. Med. Inform. Assoc., vol. 16, no. 4, pp. 471--479, 2009.Google ScholarCross Ref
- J. Braun and T. Gillespie, "Hosting the public discourse, hosting the public: When online news and social media converge," Journal. Pract., vol. 5, no. 4, pp. 383--398, 2011.Google Scholar
- A. Diaz, "Through the Google Goggles: Sociopolitical Bias in Search Engine Design," in Web search: multidisciplinary perspectives, A. Spink and M. Zimmer, Eds. Berlin: Springer, 2008, pp. 11--34.Google Scholar
- T. Gillespie, "The politics of 'platforms,'" New Media Soc., vol. 12, no. 3, pp. 347--364, May 2010.Google ScholarCross Ref
- T. Harper, "The big data public and its problems: Big data and the structural transformation of the public sphere," New Media Soc., p. 1461444816642167, 2016.Google Scholar
- P. Ladwig, A. A. Anderson, D. Brossard, D. A. Scheufele, and B. Shaw, "Narrowing the nano discourse?," Mater. Today, vol. 13, no. 5, pp. 52--54, 2010.Google ScholarCross Ref
- B. Rieder and G. Sire, "Conflicts of interest and incentives to bias: A microeconomic critique of Google's tangled position on the Web," New Media Soc., p. 1461444813481195, 2013.Google Scholar
- C. Lauckner and G. Hsieh, "The presentation of health-related search results and its impact on negative emotional outcomes," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013, pp. 333--342. Google ScholarDigital Library
- D. Kahneman, "Maps of bounded rationality: A perspective on intuitive judgment and choice," Nobel Prize Lect., vol. 8, pp. 351--401, 2002.Google Scholar
- D. A. Scheufele and D. Tewksbury, "Framing, Agenda Setting, and Priming: The Evolution of Three Media Effects Models," J. Commun., vol. 57, no. 1, pp. 9--20, Mar. 2007.Google Scholar
- J. Hullman and N. Diakopoulos, "Visualization rhetoric: Framing effects in narrative visualization," IEEE Trans. Vis. Comput. Graph., vol. 17, no. 12, pp. 2231--2240, 2011. Google ScholarDigital Library
- R. Bauman, "Genre," J. Linguist. Anthropol., vol. 9, no. 1/2, pp. 84--87, 1999.Google ScholarCross Ref
- Y.-C. Huang, H.-H. Chen, M.-L. Yeh, and Y.-C. Chung, "Case studies combined with or without concept maps improve critical thinking in hospital-based nurses: A randomized-controlled trial," Int. J. Nurs. Stud., vol. 49, no. 6, pp. 747--754, 2012.Google ScholarCross Ref
- P. Borlund, "Interactive Information Retrieval: An Introduction," J. Inf. Sci. Theory Pract., vol. 1, no. 3, pp. 12--32, Sep. 2013.Google ScholarCross Ref
- K. B. Rasmussen, "General Approaches to Data Quality and Internet-generated Data," in The Sage Handbook of online Research Methods, Fielding, Lee, and Blank, Eds. Sage, 2008.Google Scholar
- J. Creswell, Research Design - Qualitative, Quantitative, and Mixed Methods Approach, vol. 3. Sage, 2014.Google Scholar
- M. A. Carriquiry, X. Du, and G. R. Timilsina, "Second generation biofuels: Economics and policies," Spec. Sect. Renew. Energy Policy Dev., vol. 39, no. 7, pp. 4222--4234, Jul. 2011.Google Scholar
- A. B. Delshad, L. Raymond, V. Sawicki, and D. T. Wegener, "Public attitudes toward political and technological options for biofuels," Energy Policy, vol. 38, no. 7, pp. 3414--3425, Jul. 2010.Google ScholarCross Ref
- H. Longstaff, D. M. Secko, G. Capurro, P. Hanney, and T. McIntyre, "Fostering citizen deliberations on the social acceptability of renewable fuels policy: The case of advanced lignocellulosic biofuels in Canada," Biomass Bioenergy, vol. 74, pp. 103--112, Mar. 2015.Google ScholarCross Ref
- T. E. McKone et al., "Grand Challenges for Life-Cycle Assessment of Biofuels," Env. Sci Technol, vol. 45, no. 5, pp. 1751--1756, 2011.Google ScholarCross Ref
- D. M. Secko and E. Einsiedel, "The biofuels quadrilemma, public perceptions and policy," Biofuels, vol. 5, no. 3, pp. 207--209, May 2014.Google ScholarCross Ref
- D. Smith, "BFN Annual Report 2015," 2015. {Online}. Available: http://www.biofuelnet.ca/wp-content/uploads/2013/02/BFN-2015-Annual-Report_11.10.15_web.pdf. {Accessed: 25-Oct-2015}.Google Scholar
- National Resources Canada, "Biodiesel - Government Programs and Regulations | Natural Resources Canada," 2015. {Online}. Available: http://www.nrcan.gc.ca/energy/alternative-fuels/fuel-facts/biodiesel/3515. {Accessed: 06-Mar-2016}.Google Scholar
- G. Hotchkiss, T. Sherman, R. Tobin, C. Bates, and K. Brown, "Search engine results: 2010," Enq. Search Solut., pp. 1--61, 2010.Google Scholar
- K. Popper, Realism and the aim of science: From the postscript to the logic of scientific discovery. Routledge, 2013.Google ScholarCross Ref
- J. Wood, D. Badawood, J. Dykes, and A. Slingsby, "BallotMaps: Detecting name bias in alphabetically ordered ballot papers," IEEE Trans. Vis. Comput. Graph., vol. 17, no. 12, pp. 2384--2391, 2011. Google ScholarDigital Library
- E.-J. Wagenmakers, R. Wetzels, D. Borsboom, H. L. J. van der Maas, and R. A. Kievit, "An Agenda for Purely Confirmatory Research," Perspect. Psychol. Sci., vol. 7, no. 6, pp. 632--638, Nov. 2012.Google ScholarCross Ref
- M. L. Berger, M. Mamdani, D. Atkins, and M. L. Johnson, "Good Research Practices for Comparative Effectiveness Research: Defining, Reporting and Interpreting Nonrandomized Studies of Treatment Effects Using Secondary Data Sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report-Part I," Value Health, vol. 12, no. 8, pp. 1044--1052, Nov. 2009.Google Scholar
- A. Collins, J. Greeno, L. Resnick, B. Berliner, and R. Calfee, "Cognition and learning," B Berl. R Calfee Handb. Educ. Psychol. N. Y. Simon Shuster MacMillan, 1992.Google Scholar
- E. M. Meyers, "Losses and gains in collaborative search: Insights from the Middle School Classroom," GROUP'10 Novemb. 7--12 2010 Sanibel Isl. FL USA, 2010.Google Scholar
- D. M. Levy and C. C. Marshall, "Going digital: a look at assumptions underlying digital libraries," Commun. ACM, vol. 38, no. 4, pp. 77--84, Apr. 1995. Google ScholarDigital Library
- P. Dillenbourg and M. Baker, "Negotiation spaces in human-computer collaborative learning," in Proceedings of the International Conference on Cooperative Systems, 1996, pp. 12--14.Google Scholar
- J. Foster, Ed., Collaborative information behavior: user engagement and communication sharing. Hershey PA: Information Science Reference, 2010.Google ScholarCross Ref
- P. Hansen, C. Shah, and C.-P. Klas, Eds., Collaborative Information Seeking. Cham: Springer International Publishing, 2015. Google ScholarDigital Library
- M. R. Morris, "Collaborative search revisited," in Proceedings of the 2013 conference on Computer supported cooperative work, 2013, pp. 1181--1192. Google ScholarDigital Library
- M. B. Twidale, D. M. Nichols, and C. D. Paice, "Browsing is a collaborative process," Inf. Process. Manag., vol. 33, no. 6, pp. 761--783, 1997. Google ScholarDigital Library
- J. D. Novak and A. J. Cañas, "The theory underlying concept maps and how to construct and use them," Fla. Inst. Hum. Mach. Cogn., vol. 2008, 2008.Google Scholar
- J. D. Novak, Learning how to learn. Cambridge {Cambridgeshire}?; New York: Cambridge University Press, 1984.Google ScholarCross Ref
Index Terms
- Making Sense of Conflicting Science Information: Exploring Bias in the Search Engine Result Page
Recommendations
Missing Information in the Classroom and Misinformed by the Crowd: Developing Analytical Skills for Understanding Online Controversies
CHIIR '17: Proceedings of the 2017 Conference on Conference Human Information Interaction and RetrievalAt the heart of this research is the role freedom of speech and freedom of access plays in the information era. Prior research found that while our access to information is rising, so is our level of spreading misinformation. This research will build on ...
A Study of Snippet Length and Informativeness: Behaviour, Performance and User Experience
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information RetrievalThe design and presentation of a Search Engine Results Page (SERP) has been subject to much research. With many contemporary aspects of the SERP now under scrutiny, work still remains in investigating more traditional SERP components, such as the result ...
Search Interfaces and Learning about Controversial Topics
SALMM '19: Proceedings of the 1st International Workshop on Search as Learning with Multimedia InformationSearch engine results pages (SERPs) are a frequent gateway to Internet content. Prior research has extensively documented strong effects of SERPs (e.g. rank order or the spatial distribution of the results) on users' attention to and selection of ...
Comments