Skip to main content
SpringerLink
Log in
Book cover

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments pp 96–138Cite as

Crowdsourcing for Information Visualization: Promises and Pitfalls

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10264))

Abstract

Crowdsourcing offers great potential to overcome the limitations of controlled lab studies. To guide future designs of crowdsourcing-based studies for visualization, we review visualization research that has attempted to leverage crowdsourcing for empirical evaluations of visualizations. We discuss six core aspects for successful employment of crowdsourcing in empirical studies for visualization – participants, study design, study procedure, data, tasks, and metrics & measures. We then present four case studies, discussing potential mechanisms to overcome common pitfalls. This chapter will help the visualization community understand how to effectively and efficiently take advantage of the exciting potential crowdsourcing has to offer to support empirical visualization research.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.wired.com/2006/06/crowds last accessed 14 Jun 2017.

  2. 2.

    We adopt this terminology, which means a single self-contained task, from Amazon Mechanical Turk.

  3. 3.

    https://medium.com/@silberman/stop-citing-ross-et-al-2010-who-are-the-crowdworkers-b3b9b1e8d300 last accessed 14 Jun 2017.

  4. 4.

    http://demographics.mturk-tracker.com last accessed 14 Jun 2017.

  5. 5.

    http://www.quizrevolution.com/act101820/mini/go/ last accessed 14 Jun 2017, http://perceptualedge.com/files/GraphDesignIQ.html last accessed 14 Jun 2017.

  6. 6.

    http://www.colourblindawareness.org/colour-blindness/types-of-colour-blindness last accessed 14 Jun 2017.

  7. 7.

    http://www.color-blindness.com/color-blindness-tests last accessed 14 Jun 2017.

  8. 8.

    https://fold.it/portal last accessed 14 Jun 2017.

  9. 9.

    https://github.com/codementum/experimentr last accessed 14 Jun 2017.

  10. 10.

    http://archive.ics.uci.edu/ml last accessed 14 Jun 2017.

References

  1. Adams, F.M., Osgood, C.E.: A cross-cultural study of the affective meanings of color. J. Cross Cult. Psychol. 4(2), 135–156 (1973)

    Article  Google Scholar 

  2. Aigner, W., Hoffmann, S., Rind, A.: EvalBench: a software library for visualization evaluation. Comput. Graph. Forum 32(3pt1), 41–50 (2013)

    Article  Google Scholar 

  3. Aigner, W., Miksch, S., Schumann, H., Tominski, C.: Visualization of Time-Oriented Data. Human-Computer Interaction. Springer, London (2011). doi:10.1007/978-0-85729-079-3

    Book  Google Scholar 

  4. Albuquerque, G., Lowe, T., Magnor, M.: Synthetic generation of high-dimensional datasets. IEEE Trans. Vis. Comput. Graph. 17(12), 2317–2324 (2011)

    Article  Google Scholar 

  5. Alsallakh, B., Micallef, L., Aigner, W., Hauser, H., Miksch, S., Rodgers, P.: Visualizing sets and set-typed data: state-of-the-art and future challenges. In: Eurographics conference on Visualization (EuroVis)-State of The Art Reports, pp. 1–21 (2014)

    Google Scholar 

  6. Alvarez-Garcia, S., Baeza-Yates, R., Brisaboa, N.R., Larriba-Pey, J., Pedreira, O.: Graphgen: a tool for automatic generation of multipartite graphs from arbitrary data. In: 2012 Eighth Latin American Web Congress (LA-WEB), pp. 87–94. IEEE (2012)

    Google Scholar 

  7. Álvarez-García, S., Baeza-Yates, R., Brisaboa, N.R., Larriba-Pey, J.L., Pedreira, O.: Automatic multi-partite graph generation from arbitrary data. J. Syst. Softw. 94, 72–86 (2014)

    Article  Google Scholar 

  8. Amar, R., Eagan, J., Stasko, J.: Low-level components of analytic activity in information visualization. In: IEEE Symposium on Information Visualization (INFOVIS 2005), pp. 111–117. IEEE (2005)

    Google Scholar 

  9. Andrews, K., Kasanicka, J.: A comparative study of four hierarchy browsers using the hierarchical visualisation testing environment (HVTE). In: 11th International Conference Information Visualization (IV 2007), pp. 81–86. IEEE (2007)

    Google Scholar 

  10. Andrienko, G., Andrienko, N.: Privacy issues in geospatial visual analytics. In: Gartner, G., Ortag, F. (eds.) Advances in Location-Based Services. Lecture Notes in Geoinformation and Cartography. Springer, Heidelberg (2012). doi:10.1007/978-3-642-24198-7_16

    Google Scholar 

  11. Archambault, D., Purchase, H.C.: The mental map and memorability in dynamic graphs. In: Pacific Visualization Symposium (PacificVis), pp. 89–96. IEEE (2012)

    Google Scholar 

  12. Archambault, D., Purchase, H.C.: Mental map preservation helps user orientation in dynamic graphs. In: Didimo, W., Patrignani, M. (eds.) GD 2012. LNCS, vol. 7704, pp. 475–486. Springer, Heidelberg (2013). doi:10.1007/978-3-642-36763-2_42

    Chapter  Google Scholar 

  13. Bach, B., Dragicevic, P., Archambault, D., Hurter, C., Carpendale, S.: A descriptive framework for temporal data visualizations based on generalized space-time cubes. Comput. Graph. Forum (2016). http://dx.doi.org/10.1111/cgf.12804

  14. Bach, B., Spritzer, A., Lutton, E., Fekete, J.-D.: Interactive random graph generation with evolutionary algorithms. In: Didimo, W., Patrignani, M. (eds.) GD 2012. LNCS, vol. 7704, pp. 541–552. Springer, Heidelberg (2013). doi:10.1007/978-3-642-36763-2_48

    Chapter  Google Scholar 

  15. Bateman, S., Mandryk, R.L., Gutwin, C., Genest, A., McDine, D., Brooks, C.: Useful junk? The effects of visual embellishment on comprehension and memorability of charts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2573–2582. ACM (2010)

    Google Scholar 

  16. Berlin, B., Kay, P.: Basic Color Terms. University of California Press, Berkeley (1969)

    Google Scholar 

  17. Bertin, J.: Sémiologie graphique: Les diagrammes-Les réseaux-Les cartes. Gauthier-VillarsMouton & Cie (1973)

    Google Scholar 

  18. Borgo, R., Abdul-Rahman, A., Mohamed, F., Grant, P.W., Reppa, I., Floridi, L., Chen, M.: An empirical study on using visual embellishments in visualization. IEEE Trans. Vis. Comput. Graph. 18(12), 2759–2768 (2012)

    Article  Google Scholar 

  19. Boy, J., Rensink, R.A., Bertini, E., Fekete, J.D.: A principled way of assessing visualization literacy. IEEE Trans. Vis. Comput. Graph. 20(12), 1963–1972 (2014)

    Article  Google Scholar 

  20. Boy, J., Detienne, F., Fekete, J.D.: Storytelling in information visualizations: does it engage users to explore data? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1449–1458. ACM (2015)

    Google Scholar 

  21. Brehmer, M., Munzner, T.: A multi-level typology of abstract visualization tasks. IEEE Trans. Vis. Comput. Graph. 19(12), 2376–2385 (2013)

    Article  Google Scholar 

  22. Bremm, S., Von Landesberger, T., Heß, M., Fellner, D.: PCDC-on the highway to data-a tool for the fast generation of large synthetic data sets. In: EuroVis Workshop on Visual Analytics, pp. 7–11 (2012)

    Google Scholar 

  23. Brewer, C.A., MacEachren, A.M., Pickle, L.W., Herrmann, D.: Mapping mortality: evaluating color schemes for choropleth maps. Ann. Assoc. Am. Geograph. 87(3), 411–438 (1997)

    Article  Google Scholar 

  24. Brinkmann, G., McKay, B.D.: Fast generation of planar graphs. MATCH Commun. Math. Comput. Chem. 58(2), 323–357 (2007)

    MATH  MathSciNet  Google Scholar 

  25. Bristor, V.J., Drake, S.V.: Linking the language arts and content areas through visual technology. THE J. 22(2), 74–77 (1994)

    Google Scholar 

  26. Çöltekin, A., Fabrikant, S.I., Lacayo, M.: Exploring the efficiency of users’ visual analytics strategies based on sequence analysis of eye movement recordings. Int. J. Geograph. Inf. Sci. 24(10), 1559–1575 (2010)

    Article  Google Scholar 

  27. Çöltekin, A., Heil, B., Garlandini, S., Fabrikant, S.I.: Evaluating the effectiveness of interactive map interface designs: a case study integrating usability metrics with eye-movement analysis. Cartography Geogr. Inf. Sci. 36(1), 5–17 (2009)

    Article  Google Scholar 

  28. Cernea, D., Kerren, A., Ebert, A.: Detecting insight and emotion in visualization applications with a commercial EEG headset. In: SIGRAD 2011 Conference on Evaluations of Graphics and Visualization-Efficiency, Usefulness, Accessibility, Usability, pp. 53–60 (2011)

    Google Scholar 

  29. Cernea, D., Weber, C., Ebert, A., Kerren, A.: Emotion scents - a method of representing user emotions on GUI widgets. In: Proceedings of the SPIE 2013 Conference on Visualization and Data Analysis (VDA 2013). IS&T/SPIE (2013)

    Google Scholar 

  30. Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., Singh, M.: How well do line drawings depict shape? ACM Trans. Graph. 28(3), 28:1–28:9 (2009)

    Article  Google Scholar 

  31. Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper Perennia, New York (1990)

    Google Scholar 

  32. Dasgupta, A., Kosara, R.: Privacy-preserving data visualization using parallel coordinates. In: IS&T/SPIE Electronic Imaging, pp. 786800-1–786800-12. International Society for Optics and Photonics (2011)

    Google Scholar 

  33. Demiralp, Ç., Bernstein, M.S., Heer, J.: Learning perceptual kernels for visualization design. IEEE Trans. Vis. Comput. Graph. 20(12), 1933–1942 (2014)

    Article  Google Scholar 

  34. Elmqvist, N., Vande Moere, A., Jetter, H.C., Cernea, D., Reiterer, H., Jankun-Kelly, T.J.: Fluid interaction for information visualization. Inf. Vis. 10(4), 327–340 (2011)

    Article  Google Scholar 

  35. Fabrikant, S.I., Christophe, S., Papastefanou, G., Maggi, S.: Emotional response to map design aesthetics. In: 7th International Conference on Geographical Information Science, pp. 18–21 (2012)

    Google Scholar 

  36. Farrugia, M., Quigley, A.: Effective temporal graph layout: a comparative study of animation versus static display methods. Inf. Vis. 10(1), 47–64 (2011)

    Google Scholar 

  37. Fort, K., Adda, G., Cohen, K.B.: Amazon mechanical turk: gold mine or coal mine? Comput. Linguist. 37(2), 413–420 (2011)

    Article  Google Scholar 

  38. Gadiraju, U., Kawase, R., Dietze, S., Demartini, G.: Understanding malicious behavior in crowdsourcing platforms: the case of online surveys. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1631–1640. ACM (2015)

    Google Scholar 

  39. Ghani, S., Elmqvist, N.: Improving revisitation in graphs through static spatial features. In: Graphic Interface (GI 2011), pp. 737–743 (2011)

    Google Scholar 

  40. Ghani, S., Elmqvist, N., Yi, J.S.: Perception of animated node-link diagrams for dynamic graphs. Comput. Graph. Forum 31(1), 1205–1214 (2012)

    Article  Google Scholar 

  41. Giannotti, F., Pedreschi, D.: Mobility, Data Mining and Privacy: Geographic Knowledge Discovery, p. 410. Springer, Heidelberg (2008). doi:10.1007/978-3-540-75177-9

    Book  Google Scholar 

  42. van Ham, F., Rogowitz, B.: Perceptual organization in user-generated graph layouts. IEEE Trans. Vis. Comput. Graph. 14(6), 1333–1339 (2008)

    Article  Google Scholar 

  43. Haroz, S., Kosara, R., Franconeri, S.L.: Isotype visualization-working memory, performance, and engagement with pictographs. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1191–1200. ACM (2015)

    Google Scholar 

  44. Harrison, L., Yang, F., Franconeri, S., Chang, R.: Ranking visualizations of correlation using Weber’s law. IEEE Trans. Vis. Comput. Graph. 20(12), 1943–1952 (2014)

    Article  Google Scholar 

  45. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 203–212. ACM (2010)

    Google Scholar 

  46. Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform-using the example of microworkers.com. In: 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), pp. 322–329. IEEE (2011)

    Google Scholar 

  47. Isenberg, P., Zuk, T., Collins, C., Carpendale, S.: Grounded evaluation of information visualizations. In: Proceedings of the 2008 Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV 2008) pp. 6:1–6:8. ACM (2008)

    Google Scholar 

  48. Jianu, R., Rusu, A., Hu, Y., Taggart, D.: How to display group information on node-link diagrams: an evaluation. IEEE Trans. Vis. Comput. Graph. 20(11), 1530–1541 (2014)

    Article  Google Scholar 

  49. Kerren, A., Ebert, A., Meyer, J. (eds.): Human-Centered Visualization Environments. LNCS, vol. 4417. Springer, Heidelberg (2007). doi:10.1007/978-3-540-71949-6

    Google Scholar 

  50. Kerren, A., Schreiber, F.: Network visualization for integrative bioinformatics. In: Chen, M., Hofestädt, R. (eds.) Approaches in Integrative Bioinformatics, pp. 173–202. Springer, Heidelberg (2014). doi:10.1007/978-3-642-41281-3_7

    Chapter  Google Scholar 

  51. Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012)

    Article  Google Scholar 

  52. Laramee, R.S., Kosara, R.: Challenges and Unsolved Problems. In: Kerren et al. [49], pp. 231–254

    Google Scholar 

  53. Lebreton, P., Mäki, T., Skodras, E., Hupont, I., Hirth, M.: Bridging the gap between eye tracking and crowdsourcing. In: Proceedings of SPIE, vol. 9394, pp. 93940W–93940W-14 (2015)

    Google Scholar 

  54. Lee, B., Plaisant, C., Parr, C.S., Fekete, J.D., Henry, N.: Task taxonomy for graph visualization. In: Proceedings of the 2006 AVI Workshop on Beyond Time and Rrrors: Novel Evaluation Methods for Information Visualization, pp. 1–5. ACM (2006)

    Google Scholar 

  55. Li, H., Moacdieh, N.: Is “chart junk” useful? An extended examination of visual embellishment. Proc. Hum. Factors Ergon. Soc. Annual Meeting 58(1), 1516–1520 (2014)

    Article  Google Scholar 

  56. Light, A., Bartlein, P.J.: The end of the rainbow? Color schemes for improved data graphics. EOS 85(40), 385–391 (2004)

    Article  Google Scholar 

  57. Mackay, W.E., Appert, C., Beaudouin-Lafon, M., Chapuis, O., Du, Y., Fekete, J.D., Guiard, Y.: Touchstone: exploratory design of experiments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1425–1434. ACM (2007)

    Google Scholar 

  58. Maggi, S., Fabrikant, S.: Embodied decision making with animations. In: Proceedings of International Conference on Geographic Information Science 2014 (2014)

    Google Scholar 

  59. Maggi, S., Fabrikant, S.I.: Triangulating eye movement data of animated displays. In: ET4S@GIScience, pp. 27–31 (2014)

    Google Scholar 

  60. Maggi, S., Fabrikant, S.I., Imbert, J.P., Hurter, C.: How do display design and user characteristics matter in animations? An empirical study with air traffic control displays. Cartographica 51(1), 25–37 (2016)

    Article  Google Scholar 

  61. Mahyar, N., Kim, S.H., Kwon, B.C.: Towards a taxonomy for evaluating user engagement in information visualization. In: Workshop on Personal Visualization: Exploring Everyday Life (2015)

    Google Scholar 

  62. Marriott, K., Purchase, H., Wybrow, M., Goncu, C.: Memorability of visual features in network diagrams. IEEE Trans. Vis. Comput. Graph. 18(12), 2477–2485 (2012)

    Article  Google Scholar 

  63. Martin, D.: Doing Psychology Experiments, 7th edn. Thomson Wadsworth, Belmont (2008)

    Google Scholar 

  64. Martin, D., Hanrahan, B.V., O’Neill, J., Gupta, N.: Being a turker. In: Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 224–235. ACM (2014)

    Google Scholar 

  65. McCloud, S.: Understanding Comics: The Invisible Art. HarperPerennial, New York (1994)

    Google Scholar 

  66. McGee, F., Dingliana, J.: An empirical study on the impact of edge bundling on user comprehension of graphs. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, pp. 620–627. ACM (2012)

    Google Scholar 

  67. Micallef, L., Dragicevic, P., Fekete, J.D.: Assessing the effect of visualizations on bayesian reasoning through crowdsourcing. IEEE Trans. Vis. Comput. Graph. 18(12), 2536–2545 (2012)

    Article  Google Scholar 

  68. Monreale, A., Andrienko, G.L., Andrienko, N.V., Giannotti, F., Pedreschi, D., Rinzivillo, S., Wrobel, S.: Movement data anonymity through generalization. Trans. Data Priv. 3(2), 91–121 (2010)

    MathSciNet  Google Scholar 

  69. Munzner, T.: A nested model for visualization design and validation. IEEE Trans. Vis. Comput. Graph. 15(6), 921–928 (2009)

    Article  Google Scholar 

  70. Okoe, M., Jianu, R.: Graphunit: evaluating interactive graph visualizations using crowdsourcing. Comput. Graph. Forum 34(3), 451–460 (2015)

    Article  Google Scholar 

  71. Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)

    Article  Google Scholar 

  72. Pandey, A.V., Rall, K., Satterthwaite, M.L., Nov, O., Bertini, E.: How deceptive are deceptive visualizations? An empirical analysis of common distortion techniques. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1469–1478. ACM (2015)

    Google Scholar 

  73. Papadopoulos, C., Gutenko, I., Kaufman, A.: VEEVVIE: visual explorer for empirical visualization, VR and interaction experiments. IEEE Trans. Vis. Comput. Graph. 22(1), 111–120 (2016)

    Article  Google Scholar 

  74. Peck, E.M.M., Yuksel, B.F., Ottley, A., Jacob, R.J., Chang, R.: Using fNIRS brain sensing to evaluate information visualization interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 473–482. ACM (2013)

    Google Scholar 

  75. Plaisant, C.: The challenge of information visualization evaluation. In: Proceedings of the Working Conference on Advanced Visual Interfaces (AVI 2004), pp. 109–116. ACM (2004)

    Google Scholar 

  76. Purchase, H.: Which aesthetic has the greatest effect on human understanding? In: DiBattista, G. (ed.) GD 1997. LNCS, vol. 1353, pp. 248–261. Springer, Heidelberg (1997). doi:10.1007/3-540-63938-1_67

    Chapter  Google Scholar 

  77. Purchase, H.C.: Experimental Human-Computer Interaction: A Practical Guide with Visual Examples. Cambridge University Press, Cambridge (2012)

    Book  Google Scholar 

  78. Ross, J., Irani, L., Silberman, M., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers? Shifting demographics in mechanical turk. In: CHI 2010 Extended Abstracts on Human Factors in Computing Systems, pp. 2863–2872. ACM (2010)

    Google Scholar 

  79. Saket, B., Scheidegger, C., Kobourov, S.: Towards understanding enjoyment and flow in information visualization. In: EuroVis. The Eurographics Association (Short Paper) (2015)

    Google Scholar 

  80. Saket, B., Scheidegger, C., Kobourov, S.: Comparing node-link and node-link-group visualizations from an enjoyment perspective. Comput. Graph. Forum 35(3), 41–50 (2016)

    Article  Google Scholar 

  81. Saket, B., Scheidegger, C., Kobourov, S.G., Börner, K.: Map-based visualizations increase recall accuracy of data. Comput. Graph. Forum 34(3), 441–450. http://dx.doi.org/10.1111/cgf.12656

  82. Sakshaug, J.W., Raghunathan, T.E.: Synthetic data for small area estimation. In: Domingo-Ferrer, J., Magkos, E. (eds.) PSD 2010. LNCS, vol. 6344, pp. 162–173. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15838-4_15

    Chapter  Google Scholar 

  83. Sakshaug, J.W., Raghunathan, T.E.: Generating synthetic data to produce public-use microdata for small geographic areas based on complex sample survey data with application to the national health interview survey. J. Appl. Stat. 41(10), 2103–2122 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  84. Sakshaug, J.W., Raghunathan, T.E.: Nonparametric generation of synthetic data for small geographic areas. In: Domingo-Ferrer, J. (ed.) PSD 2014. LNCS, vol. 8744, pp. 213–231. Springer, Cham (2014). doi:10.1007/978-3-319-11257-2_17

    Google Scholar 

  85. Shneiderman, B.: The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE Symposium on Visual Languages, Boulder, Colorado, USA, 3–6 September 1996, pp. 336–343. IEEE Computer Society (1996)

    Google Scholar 

  86. Tanahashi, Y., Ma, K.L.: Stock lamp: an engagement-versatile visualization design. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 595–604. ACM (2015)

    Google Scholar 

  87. Theodoridis, Y., Silva, J.R.O., Nascimento, M.A.: On the generation of spatiotemporal datasets. In: Güting, R.H., Papadias, D., Lochovsky, F. (eds.) SSD 1999. LNCS, vol. 1651, pp. 147–164. Springer, Heidelberg (1999). doi:10.1007/3-540-48482-5_11

    Chapter  Google Scholar 

  88. Valiati, E.R., Pimenta, M.S., Freitas, C.M.: A taxonomy of tasks for guiding the evaluation of multidimensional visualizations. In: Proceedings of the 2006 AVI Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization, pp. 1–6. ACM (2006)

    Google Scholar 

  89. Vande Moere, A., Tomitsch, M., Wimmer, C., Christoph, B., Grechenig, T.: Evaluating the effect of style in information visualization. IEEE Trans. Vis. Comput. Graph. 18(12), 2739–2748 (2012)

    Article  Google Scholar 

  90. Wainer, H.: A test of graphicacy in children. Appl. Psychol. Measure. 4(3), 331–340 (1980)

    Article  Google Scholar 

  91. Walny, J., Huron, S., Carpendale, S.: An exploratory study of data sketching for visual representation. Comput. Graph. Forum 34(3), 231–240 (2015)

    Article  Google Scholar 

  92. Wang, B., Ruchikachorn, P., Mueller, K.: SketchPadN-D: WYDIWYG sculpting and editing in high-dimensional space. IEEE Trans. Vis. Comput. Graph. 19(12), 2060–2069 (2013)

    Article  Google Scholar 

  93. Ware, C.: Information Visualization: Preception for Design, 3rd edn. Elsevier, Amsterdam (2013)

    Google Scholar 

  94. Ware, C., Mitchell, P.: Visualizing graphs in three dimensions. ACM Trans. Appl. Percept. 5(1), 2:1–2:15 (2008)

    Article  Google Scholar 

  95. Wilkening, J., Fabrikant, S.I.: How users interact with a 3d geo-browser under time pressure. Cartography Geogr. Inf. Sci. 40(1), 40–52 (2013)

    Article  Google Scholar 

  96. Xu, P., Ehinger, K.A., Zhang, Y., Finkelstein, A., Kulkarni, S.R., Xiao, J.: TurkerGaze: crowdsourcing saliency with webcam based eye tracking. CoRR abs/1504.06755 (2015)

    Google Scholar 

  97. Yang, H., Li, Y., Zhou, M.X.: Understand users’ comprehension and preferences for composing information visualizations. ACM Trans. Comput. Hum. Interact. 21(1), 6:1–6:30 (2014)

    Article  Google Scholar 

  98. Ying, X., Wu, X.: Graph generation with prescribed feature constraints. In: SDM, vol. 9, pp. 966–977. SIAM (2009)

    Google Scholar 

  99. Ziemkiewicz, C., Kosara, R.: Preconceptions and individual differences in understanding visual metaphors. Comput. Graph. Forum 28(3), 911–918 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. King’s College London, London, UK

    Rita Borgo

  2. Microsoft Research, Redmond, USA

    Bongshin Lee

  3. Microsoft Research - Inria, Paris, France

    Benjamin Bach

  4. University of Zurich, Zurich, Switzerland

    Sara Fabrikant

  5. City University London, London, UK

    Radu Jianu

  6. Linnaeus University, Växjö, Sweden

    Andreas Kerren

  7. University of Arizona, Tucson, USA

    Stephen Kobourov

  8. Luxembourg Institute of Science and Technology, Esch-sur-Alzette, Luxembourg

    Fintan McGee

  9. Helsinki Institute for Information Technology, Aalto, Finland

    Luana Micallef

  10. Darmstadt University, Darmstadt, Germany

    Tatiana von Landesberger & Katrin Ballweg

  11. University Trier, Trier, Germany

    Stephan Diehl

  12. Juji, Saratoga, USA

    Michelle Zhou

  13. Swansea University, Swansea, UK

    Paolo Simonetto

Authors
  1. Rita Borgo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bongshin Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin Bach
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sara Fabrikant
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Radu Jianu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andreas Kerren
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Stephen Kobourov
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Fintan McGee
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Luana Micallef
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tatiana von Landesberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Katrin Ballweg
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Stephan Diehl
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Paolo Simonetto
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Michelle Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rita Borgo .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Swansea University, Swansea, United Kingdom

    Daniel Archambault

  2. University of Glasgow, Glasgow, United Kingdom

    Helen Purchase

  3. Modellierung adaptiver Systeme, Universität Duisburg-Essen, Essen, Germany

    Tobias Hoßfeld

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Borgo, R. et al. (2017). Crowdsourcing for Information Visualization: Promises and Pitfalls. In: Archambault, D., Purchase, H., Hoßfeld, T. (eds) Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Lecture Notes in Computer Science(), vol 10264. Springer, Cham. https://doi.org/10.1007/978-3-319-66435-4_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66435-4_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66434-7

  • Online ISBN: 978-3-319-66435-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation