Skip to main content
Log in

Examining interaction techniques in data visualization authoring tools from the perspective of goals and human cognition: a survey

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

We review the state-of-the-art interaction techniques of visualization authoring tools. The visualization tools tend to help users in the creation, exploration, or presentation of visualizations. Also, they allow users to craft expressive designs or extract data from visualizations. The review presents the interaction techniques integrated into the tools for those mentioned above five high-level goals. We cover each goal’s tools and summarize how a sequence in the independent interaction techniques leads to the goal. We also discuss how well researchers had evaluated the usability and intuitiveness of interaction techniques. We aimed to reflect on the strengths and weaknesses of the evaluations. To that end, from the perspective of human cognition, we reviewed the goals, procedures, and findings of evaluations. Principally, human cognition is engaged when they perform tasks in a tool. The interaction techniques bridge the gap between human cognition and the goals they want to achieve from the tool. To sum up, in this review, we present a novel triad ‘goals-interaction techniques-cognition’ taxonomy. Besides, the review suggests the need for further work to enhance tools and understand users.

Graphic Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Alper B, Riche NH, Chevalier F, Boy J, Sezgin M (2017) Visualization literacy at elementary school. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 5485–5497

  • Amar R, Stasko J (2004) A knowledge task-based framework for design and evaluation of information visualizations. In: Proceedings of IEEE Symposium on Information Visualization, IEEE, pp 143–150

  • Amini F, Riche NH, Lee B, Monroy-Hernandez A, Irani P (2017) Authoring data-driven videos with dataclips. IEEE Trans Visual Comput Graph 23(1):501–510

    Article  Google Scholar 

  • Amini F, Brehmer M, Bolduan G, Elmer C, Wiederkehr B (2018) Evaluating data-driven stories and storytelling tools. Data-driven storytelling. AK Peters/CRC Press, Boca Raton, pp 249–286

    Chapter  Google Scholar 

  • André P, Wilson ML, Russell A, Smith DA, Owens A, Schraefel M (2007) Continuum: designing timelines for hierarchies, relationships and scale. In: Proceedings of 20th Annual ACM Symposium on User Interface Software and Technology, pp 101–110

  • Attride-Stirling J (2001) Thematic networks: an analytic tool for qualitative research. Qual Res 1(3):385–405

    Article  Google Scholar 

  • Bishop F, Zagermann J, Pfeil U, Sanderson G, Reiterer H, Hinrichs U (2019) Construct-a-vis: exploring the free-form visualization processes of children. IEEE Trans Visual Comput Graph 26(1):451–460

    Google Scholar 

  • Bostock M, Ogievetsky V, Heer J (2011) \({\text{D}}^3\) data-driven documents. IEEE Trans Visual Comput Graph 17(12):2301–2309

    Article  Google Scholar 

  • Boussejra MO, Uchiki R, Takeshima Y, Matsubayashi K, Takekawa S, Makoto U, Fujishiro I (2019) aflak: Visual programming environment enabling end-to-end provenance management for the analysis of astronomical datasets. Vis Inf 3(1):1–8

    Google Scholar 

  • Brehmer M, Munzner T (2013) A multi-level typology of abstract visualization tasks. IEEE Trans Visual Comput Graph 19(12):2376–2385

    Article  Google Scholar 

  • Brehmer M, Lee B, Bach B, Riche NH, Munzner T (2017) Timelines revisited: a design space and considerations for expressive storytelling. IEEE Trans Visual Comput Graph 23(9):2151–2164

    Article  Google Scholar 

  • Browne J, Lee B, Carpendale S, Riche N, Sherwood T (2011) Data analysis on interactive whiteboards through sketch-based interaction. In: Proceedings of ACM International Conference on Interactive Tabletops and Surfaces, pp 154–157

  • Bryan C, Mishra A, Shidara H, Ma KL (2020) Analyzing gaze behavior for text-embellished narrative visualizations under different task scenarios. Vis Inf 4(3):41–50

    Google Scholar 

  • Cao R, Dey S, Cunningham A, Walsh J, Smith RT, Zucco JE, Thomas BH (2020) Examining the use of narrative constructs in data videos. Vis Inf 4(1):8–22

    Google Scholar 

  • Ceneda D, Gschwandtner T, Miksch S (2019) You get by with a little help: the effects of variable guidance degrees on performance and mental state. Vis Inf 3(4):177–191

    Google Scholar 

  • Chao WO, Munzner T, Panne M (2010) Napkinvis: Rapid pen-centric authoring of improvisational visualizations. In: Proceedings of IEEE VIS

  • Chen Z, Wang Y, Sun T, Gao X, Chen W, Pan Z, Qu H, Wu Y (2017) Exploring the design space of immersive urban analytics. Vis Inf 1(2):132–142

    Google Scholar 

  • Chen Z, Su Y, Wang Y, Wang Q, Qu H, Wu Y (2019) Marvist: Authoring glyph-based visualization in mobile augmented reality. IEEE Trans Visual Comput Graph 26(8):2645–2658

    Article  Google Scholar 

  • Choi J, Park DG, Wong YL, Fisher E, Elmqvist N (2015) Visdock: a toolkit for cross-cutting interactions in visualization. IEEE Trans Visual Comput Graph 21(9):1087–1100

    Article  Google Scholar 

  • Chotisarn N, Lu J, Ma L, Xu J, Meng L, Lin B, Xu Y, Luo X, Chen W (2020) Bubble storytelling with automated animation: a brexit hashtag activism case study. Journal of Visualization

  • Crapo AW, Waisel LB, Wallace WA, Willemain TR (2000) Visualization and the process of modeling: a cognitive-theoretic view. In: Proceedings of Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 218–226

  • Cui W, Zhang X, Wang Y, Huang H, Chen B, Fang L, Zhang H, Lou JG, Zhang D (2019) Text-to-viz: Automatic generation of infographics from proportion-related natural language statements. IEEE Trans Visual Comput Graphics 26(1):906–916

    Article  Google Scholar 

  • Deng H, Wang X, Guo Z, Decker A, Duan X, Wang C, Ambrose G, Abbott K (2019) Performancevis: Visual analytics of student performance data from an introductory chemistry course. Vis Inf 3(4):166–176

    Google Scholar 

  • Dimara E, Perin C (2019) What is interaction for data visualization? IEEE Trans Visual Comput Graph 26(1):119–129

    Article  Google Scholar 

  • Elting LS, Martin CG, Cantor SB, Rubenstein EB (1999) Influence of data display formats on physician investigators’ decisions to stop clinical trials: prospective trial with repeated measures. Br Med J 318(7197):1527–1531

    Article  Google Scholar 

  • Ghosh A, Nashaat M, Miller J, Quader S, Marston C (2018) A comprehensive review of tools for exploratory analysis of tabular industrial datasets. Vis Inf 2(4):235–253

    Google Scholar 

  • Green TM, Ribarsky W, Fisher B (2008) Visual analytics for complex concepts using a human cognition model. In: Proceedings of IEEE Symposium on Visual Analytics Science and Technology, IEEE, pp 91–98

  • Heer J, Viégas FB, Wattenberg M (2007) Voyagers and voyeurs: supporting asynchronous collaborative information visualization. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 1029–1038

  • Hu T, Zheng H, Liang C, Zhu S, Imirzian N, Zhang Y, Wang C, Hughes DP, Chen DZ (2020) Antvis: A web-based visual analytics tool for exploring ant movement data. Vis Inf 4(1):58–70

    Google Scholar 

  • Isenberg T, Isenberg P, Chen J, Sedlmair M, Möller T (2013) A systematic review on the practice of evaluating visualization. IEEE Trans Vis Comput Graph 19(12):2818–2827

    Article  Google Scholar 

  • Jung D, Kim W, Song H, Hwang Ji, Lee B, Kim B, Seo J (2017) Chartsense: Interactive data extraction from chart images. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 6706–6717

  • Kim NW, Schweickart E, Liu Z, Dontcheva M, Li W, Popovic J, Pfister H (2017a) Data-driven guides: supporting expressive design for information graphics. IEEE Trans Vis Comput Graph 23(1):491–500

    Article  Google Scholar 

  • Kim NW, Bach B, Im H, Schriber S, Gross M, Pfister H (2018) Visualizing nonlinear narratives with story curves. IEEE Trans Vis Comput Graph 24(1):595–604

    Article  Google Scholar 

  • Kim NW, Im H, Riche NH, Wang A, Gajos KZ, Pfister H (2019a) Dataselfie: Empowering people to design personalized visuals to represent their data. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems

  • Kim NW, Riche NH, Bach B, Xu G, Brehmer M, Hinckley K, Pahud M, Xia H, McGuffin MJ, Pfister H (2019b) Datatoon: Drawing data comics about dynamic networks with pen + touch interaction. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems

  • Kim YS, Reinecke K, Hullman J (2017b) Explaining the gap: Visualizing one’s predictions improves recall and comprehension of data. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 1375–1386

  • Klein G, Hoffman RR (2008) Macrocognition, mental models, and cognitive task analysis methodology. Naturalistic Decision Making and Macrocognition pp 57–80

  • Koytek P, Perin C, Vermeulen J, André E, Carpendale S (2018) Mybrush: brushing and linking with personal agency. IEEE Trans Vis Comput Graph 24(1):605–615

    Article  Google Scholar 

  • Lam H (2008) A framework of interaction costs in information visualization. IEEE Trans Vis Comput Graph 14(6):1149–1156

    Article  Google Scholar 

  • Lam H, Bertini E, Isenberg P, Plaisant C, Carpendale S (2011) Seven guiding scenarios for information visualization evaluation

  • Lee B, Kazi RH, Smith G (2013) Sketchstory: telling more engaging stories with data through freeform sketching conference on human factors in computing systems. IEEE Trans Vis Comput Graph 19(12):2416–2425

    Article  Google Scholar 

  • Lee B, Isaacs K, Szafir DA, Marai GE, Turkay C, Tory M, Carpendale S, Endert A (2019) Broadening intellectual diversity in visualization research papers. IEEE Comput Graph Appl 39(4):78–85

    Article  Google Scholar 

  • Li D, Mei H, Shen Y, Su S, Zhang W, Wang J, Zu M, Chen W (2018) Echarts: a declarative framework for rapid construction of web-based visualization. Vis Inf 2(2):136–146

    Google Scholar 

  • Liu Z, Stasko J (2010) Mental models, visual reasoning and interaction in information visualization: A top-down perspective. IEEE Trans Vis Comput Graph 16(6):999–1008

    Article  Google Scholar 

  • Liu Z, Thompson J, Wilson A, Dontcheva M, Delorey J, Grigg S, Kerr B, Stasko J (2018) Data illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, p 123

  • Lu Z, Fan M, Wang Y, Zhao J, Annett M, Wigdor D (2019) Inkplanner: supporting prewriting via intelligent visual diagramming. IEEE Trans Vis Comput Graph 25(1):277–287

    Article  Google Scholar 

  • Mackinlay J, Hanrahan P, Stolte C (2007) Show me: automatic presentation for visual analysis. IEEE Trans Vis Comput Graph 13(6):1137–1144

    Article  Google Scholar 

  • Mei H, Ma Y, Wei Y, Chen W (2017) The design space of construction tools for information visualization: A survey. J Vis Lang Comput 44:120–132

    Article  Google Scholar 

  • Mei H, Chen W, Ma Y, Guan H, Hu W (2018) Viscomposer: a visual programmable composition environment for information visualization. Vis Inf 2(1):71–81

    Google Scholar 

  • Mei H, Guan H, Xin C, Wen X, Chen W (2020) Datav: data visualization on large high-resolution displays. Vis Inf 4(3):12–23

    Google Scholar 

  • Méndez GG, Hinrichs U, Nacenta MA (2017) Bottom-up vs. top-down: Trade-offs in efficiency, understanding, freedom and creativity with infovis tools. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 841–852

  • Méndez GG, Nacenta MA, Hinrichs U (2018) Considering agency and data granularity in the design of visualization tools. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems

  • Nacenta MA, Méndez GG (2017) ivolver: A visual language for constructing visualizations from in-the-wild data. In: Proceedings of ACM International Conference on Interactive Surfaces and Spaces, pp 438–441

  • Norman D (2013) The design of everyday things: Revised and expanded edition. Basic books

  • Pan J, Chen W, Zhao X, Zhou S, Zeng W, Zhu M, Chen J, Fu S, Wu Y (to appear) Exemplar-based layout fine-tuning for node-link diagrams. IEEE Transactions on Visualization and Computer Graphics. In Proceedings of IEEE VIS 2020

  • Pantazos K, Lauesen S (2012) Constructing visualizations with infovis tools-an evaluation from a user perspective. GRAPP/IVAPP 2012:731–736

    Google Scholar 

  • Patterson RE, Blaha LM, Grinstein GG, Liggett KK, Kaveney DE, Sheldon KC, Havig PR, Moore JA (2014) A human cognition framework for information visualization. Comput Graph 42:42–58

    Article  Google Scholar 

  • Pike WA, Stasko J, Chang R, O’connell TA (2009) The science of interaction. Inf Vis 8(4):263–274

    Article  Google Scholar 

  • Ren D, Höllerer T, Yuan X (2014) ivisdesigner: expressive interactive design of information visualizations. IEEE Trans Vis Comput Graph 20(12):2092–2101

    Article  Google Scholar 

  • Ren D, Lee B, Brehmer M (2019) Charticulator: interactive construction of bespoke chart layouts. IEEE Trans Vis Comput Graph 25(1):789–799

    Article  Google Scholar 

  • Roberts JC, Ritsos PD, Jackson JR, Headleand C (2017) The explanatory visualization framework: an active learning framework for teaching creative computing using explanatory visualizations. IEEE Trans Vis Comput Graph 24(1):791–801

    Article  Google Scholar 

  • Romat H, Appert C, Pietriga E (2019a) Expressive authoring of node-link diagrams with graphies. IEEE Transactions on Visualization and Computer Graphics

  • Romat H, Henry Riche N, Hinckley K, Lee B, Appert C, Pietriga E, Collins C (2019b) Activeink: (th) inking with data. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 1–13

  • Saket B, Kim H, Brown ET, Endert A (2017) Visualization by demonstration: an interaction paradigm for visual data exploration. IEEE Trans Vis Comput Graph 23(1):331–340

    Article  Google Scholar 

  • Sarracino J, Barrios-Arciga O, Zhu J, Marcus N, Lerner S, Wiedermann B (2017) User-guided synthesis of interactive diagrams. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 195–207

  • Satyanarayan A, Heer J (2014a) Authoring narrative visualizations with ellipsis. Comput Graph Forum 33(3):361–370

    Article  Google Scholar 

  • Satyanarayan A, Heer J (2014b) Lyra: an interactive visualization design environment. Comput Graph Forum 33(3):351–360

    Article  Google Scholar 

  • Satyanarayan A, Russell R, Hoffswell J, Heer J (2016) Reactive vega: a streaming dataflow architecture conference on human factors in computing systems for declarative interactive visualization. IEEE Trans Vis Comput Graph 22(1):659–668

    Article  Google Scholar 

  • Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J (2017) Vega-lite: a grammar of interactive graphics. IEEE Trans Vis Comput Graph 23(1):341–350

    Article  Google Scholar 

  • Satyanarayan A, Lee B, Ren D, Heer J, Stasko J, Thompson J, Brehmer M, Liu Z (2019) Critical reflections on visualization authoring systems. IEEE Trans Vis Comput Graph 26(1):461–471

    Google Scholar 

  • Scaife M, Rogers Y (1996) External cognition: how do graphical representations work? Int J Hum Comput Stud 45(2):185–213

    Article  Google Scholar 

  • Shen E, Li S, Cai X, Zeng L, Wang W (2014) Sketch-based interactive visualization: a survey. J Vis 17(4):275–294

    Article  Google Scholar 

  • Shneiderman B (1996) The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings 1996 IEEE Symposium on Visual Languages, IEEE, pp 336–343

  • Shu X, Wu J, Wu X, Liang H, Cui W, Wu Y, Qu H (2020) Dancingwords: exploring animated word clouds to tell stories. Journal of Visualization

  • Shu X, Wu A, Tang J, Bach B, Wu Y, Qu H (to appear) What makes a data-gif understandable? IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE VIS 2020)

  • Sicat R, Li J, Choi J, Cordeil M, Jeong WK, Bach B, Pfister H (2019) Dxr: a toolkit for building immersive data visualizations. IEEE Trans Vis Comput Graph 25(1):715–725

    Article  Google Scholar 

  • Tang J, Yu L, Tang T, Shu X, Ying L, Zhou Y, Ren P, Wu Y (to appeara) Narrative transitions in data videos. In: Proceedings of IEEE VIS 2020 Short Papers, IEEE

  • Tang T, Rubab S, Lai J, Cui W, Yu L, Wu Y (2019) istoryline: effective convergence to hand-drawn storylines. IEEE Trans Vis Comput Graph 25(1):769–778

    Article  Google Scholar 

  • Tang T, Li R, Wu X, Liu S, Knittel J, Koch S, Ertl T, Yu L, Ren P, Wu Y (to appearb) Plotthread: Creating expressive storyline visualizations using reinforcement learning. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE VIS 2020)

  • Tong C, Roberts R, Borgo R, Walton S, Laramee RS, Wegba K, Lu A, Wang Y, Qu H, Luo Q et al (2018) Storytelling and visualization: an extended survey. Information 9(3):65

    Article  Google Scholar 

  • Tory M, Moller T (2004) Human factors in visualization research. IEEE Trans Vis Comput Graph 10(1):72–84

    Article  Google Scholar 

  • Tversky B, Agrawala M, Heiser J, Lee P, Hanrahan P, Phan D, Stolte C, Daniel MP (2006) Cognitive design principles for automated generation of visualizations. From Research to Cognitive Technology, Applied Spatial Cognition, pp 53–75

    Google Scholar 

  • Viegas FB, Wattenberg M, Van Ham F, Kriss J, McKeon M (2007) Manyeyes: a site for visualization at internet scale. IEEE Transactions on Visualization and Computer Graphics 13(6):

  • Wang Q, Li Z, Fu S, Cui W, Qu H (2019) Narvis: authoring narrative slideshows for introducing data visualization designs. IEEE Trans Vis Comput Graph 25(1):779–788

    Article  Google Scholar 

  • Wang Y, Chu X, Bao C, Zhu L, Deussen O, Chen B, Sedlmair M (2018a) Edwordle: consistency-preserving word cloud editing. IEEE Trans Vis Comput Graph 24(1):647–656

    Article  Google Scholar 

  • Wang Y, Zhang H, Huang H, Chen X, Yin Q, Hou Z, Zhang D, Luo Q, Qu H (2018b) Infonice: Easy creation of information graphics. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems

  • Weng D, Zheng C, Deng Z, Ma M, Bao J, Zheng Y, Xu M, Wu Y (to appear) Towards better bus networks: A visual analytics approach. IEEE Transactions on Visualization and Computer Graphics. In Proceedings of IEEE VIS 2020

  • Willett W, Heer J, Hellerstein J, Agrawala M (2011) Commentspace: structured support for collaborative visual analysis. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 3131–3140

  • Wongsuphasawat K, Moritz D, Anand A, Mackinlay J, Howe B, Heer J (2015) Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans Vis Comput Graph 22(1):649–658

    Article  Google Scholar 

  • Wongsuphasawat K, Qu Z, Moritz D, Chang R, Ouk F, Anand A, Mackinlay J, Howe B, Heer J (2017) Voyager 2: Augmenting visual analysis with partial view specifications. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 2648–2659

  • Xia H, Araujo B, Grossman T, Wigdor D (2016) Object-oriented drawing. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 4610–4621

  • Xia H, Henry Riche N, Chevalier F, De Araujo B, Wigdor D (2018) Dataink: Direct and creative data-oriented drawing. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems

  • Yalçin MA, Elmqvist N, Bederson BB (2016) Cognitive stages in visual data exploration. In: Proceedings of Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, pp 86–95

  • Yalcin MA, Elmqvist N, Bederson BB (2018) Keshif: rapid and expressive tabular data exploration for novices. IEEE Trans Vis Comput Graph 24(8):2339–2352

    Article  Google Scholar 

  • Ye S, Chen Z, Chu X, Wang Y, Fu S, Shen L, Zhou K, Wu Y (to appear) Shuttlespace: Exploring and analyzing movement trajectory in immersive visualization. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE VIS 2020)

  • Yi JS, ah Kang Stasko J, Y, Stasko J (2007) Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans Vis Comput Graph 13(6):1224–1231

    Article  Google Scholar 

  • Yoghourdjian V, Archambault D, Diehl S, Dwyer T, Klein K, Purchase HC, Wu HY (2018) Exploring the limits of complexity: a survey of empirical studies on graph visualisation. Vis Inf 2(4):264–282

    Google Scholar 

  • Zacks JM, Tversky B (2003) Structuring information interfaces for procedural learning. J Exp Psychol Appl 9(2):88

    Article  Google Scholar 

  • Zgraggen E, Zeleznik R, Drucker SM (2014) Panoramicdata: data analysis through pen and touch. IEEE Trans Vis Comput Graph 20(12):2112–2121

    Article  Google Scholar 

  • Zhao Z, Marr R, Elmqvist N (2015) Data comics: Sequential art for data-driven storytelling. Tech Report

  • Zhu S, Sun G, Jiang Q, Zha M, Liang R (2020) A survey on automatic infographics and visualization recommendations. Vis Inf 4(3):24–40

    Google Scholar 

Download references

Acknowledgements

The work was supported by NSFC (61761136020), NSFC-Zhejiang Joint Fund for the Integration of Industrialization and Informatization (U1609217), Zhejiang Provincial Natural Science Foundation (LR18F020001) and the 100 Talents Program of Zhejiang University. This project was also partially funded by Microsoft Research Asia.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yingcai Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rubab, S., Tang, J. & Wu, Y. Examining interaction techniques in data visualization authoring tools from the perspective of goals and human cognition: a survey. J Vis 24, 397–418 (2021). https://doi.org/10.1007/s12650-020-00705-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-020-00705-3

Keywords

Navigation