Abstract
We review the state-of-the-art interaction techniques of visualization authoring tools. The visualization tools tend to help users in the creation, exploration, or presentation of visualizations. Also, they allow users to craft expressive designs or extract data from visualizations. The review presents the interaction techniques integrated into the tools for those mentioned above five high-level goals. We cover each goal’s tools and summarize how a sequence in the independent interaction techniques leads to the goal. We also discuss how well researchers had evaluated the usability and intuitiveness of interaction techniques. We aimed to reflect on the strengths and weaknesses of the evaluations. To that end, from the perspective of human cognition, we reviewed the goals, procedures, and findings of evaluations. Principally, human cognition is engaged when they perform tasks in a tool. The interaction techniques bridge the gap between human cognition and the goals they want to achieve from the tool. To sum up, in this review, we present a novel triad ‘goals-interaction techniques-cognition’ taxonomy. Besides, the review suggests the need for further work to enhance tools and understand users.
Graphic Abstract
Similar content being viewed by others
References
Alper B, Riche NH, Chevalier F, Boy J, Sezgin M (2017) Visualization literacy at elementary school. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 5485–5497
Amar R, Stasko J (2004) A knowledge task-based framework for design and evaluation of information visualizations. In: Proceedings of IEEE Symposium on Information Visualization, IEEE, pp 143–150
Amini F, Riche NH, Lee B, Monroy-Hernandez A, Irani P (2017) Authoring data-driven videos with dataclips. IEEE Trans Visual Comput Graph 23(1):501–510
Amini F, Brehmer M, Bolduan G, Elmer C, Wiederkehr B (2018) Evaluating data-driven stories and storytelling tools. Data-driven storytelling. AK Peters/CRC Press, Boca Raton, pp 249–286
André P, Wilson ML, Russell A, Smith DA, Owens A, Schraefel M (2007) Continuum: designing timelines for hierarchies, relationships and scale. In: Proceedings of 20th Annual ACM Symposium on User Interface Software and Technology, pp 101–110
Attride-Stirling J (2001) Thematic networks: an analytic tool for qualitative research. Qual Res 1(3):385–405
Bishop F, Zagermann J, Pfeil U, Sanderson G, Reiterer H, Hinrichs U (2019) Construct-a-vis: exploring the free-form visualization processes of children. IEEE Trans Visual Comput Graph 26(1):451–460
Bostock M, Ogievetsky V, Heer J (2011) \({\text{D}}^3\) data-driven documents. IEEE Trans Visual Comput Graph 17(12):2301–2309
Boussejra MO, Uchiki R, Takeshima Y, Matsubayashi K, Takekawa S, Makoto U, Fujishiro I (2019) aflak: Visual programming environment enabling end-to-end provenance management for the analysis of astronomical datasets. Vis Inf 3(1):1–8
Brehmer M, Munzner T (2013) A multi-level typology of abstract visualization tasks. IEEE Trans Visual Comput Graph 19(12):2376–2385
Brehmer M, Lee B, Bach B, Riche NH, Munzner T (2017) Timelines revisited: a design space and considerations for expressive storytelling. IEEE Trans Visual Comput Graph 23(9):2151–2164
Browne J, Lee B, Carpendale S, Riche N, Sherwood T (2011) Data analysis on interactive whiteboards through sketch-based interaction. In: Proceedings of ACM International Conference on Interactive Tabletops and Surfaces, pp 154–157
Bryan C, Mishra A, Shidara H, Ma KL (2020) Analyzing gaze behavior for text-embellished narrative visualizations under different task scenarios. Vis Inf 4(3):41–50
Cao R, Dey S, Cunningham A, Walsh J, Smith RT, Zucco JE, Thomas BH (2020) Examining the use of narrative constructs in data videos. Vis Inf 4(1):8–22
Ceneda D, Gschwandtner T, Miksch S (2019) You get by with a little help: the effects of variable guidance degrees on performance and mental state. Vis Inf 3(4):177–191
Chao WO, Munzner T, Panne M (2010) Napkinvis: Rapid pen-centric authoring of improvisational visualizations. In: Proceedings of IEEE VIS
Chen Z, Wang Y, Sun T, Gao X, Chen W, Pan Z, Qu H, Wu Y (2017) Exploring the design space of immersive urban analytics. Vis Inf 1(2):132–142
Chen Z, Su Y, Wang Y, Wang Q, Qu H, Wu Y (2019) Marvist: Authoring glyph-based visualization in mobile augmented reality. IEEE Trans Visual Comput Graph 26(8):2645–2658
Choi J, Park DG, Wong YL, Fisher E, Elmqvist N (2015) Visdock: a toolkit for cross-cutting interactions in visualization. IEEE Trans Visual Comput Graph 21(9):1087–1100
Chotisarn N, Lu J, Ma L, Xu J, Meng L, Lin B, Xu Y, Luo X, Chen W (2020) Bubble storytelling with automated animation: a brexit hashtag activism case study. Journal of Visualization
Crapo AW, Waisel LB, Wallace WA, Willemain TR (2000) Visualization and the process of modeling: a cognitive-theoretic view. In: Proceedings of Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 218–226
Cui W, Zhang X, Wang Y, Huang H, Chen B, Fang L, Zhang H, Lou JG, Zhang D (2019) Text-to-viz: Automatic generation of infographics from proportion-related natural language statements. IEEE Trans Visual Comput Graphics 26(1):906–916
Deng H, Wang X, Guo Z, Decker A, Duan X, Wang C, Ambrose G, Abbott K (2019) Performancevis: Visual analytics of student performance data from an introductory chemistry course. Vis Inf 3(4):166–176
Dimara E, Perin C (2019) What is interaction for data visualization? IEEE Trans Visual Comput Graph 26(1):119–129
Elting LS, Martin CG, Cantor SB, Rubenstein EB (1999) Influence of data display formats on physician investigators’ decisions to stop clinical trials: prospective trial with repeated measures. Br Med J 318(7197):1527–1531
Ghosh A, Nashaat M, Miller J, Quader S, Marston C (2018) A comprehensive review of tools for exploratory analysis of tabular industrial datasets. Vis Inf 2(4):235–253
Green TM, Ribarsky W, Fisher B (2008) Visual analytics for complex concepts using a human cognition model. In: Proceedings of IEEE Symposium on Visual Analytics Science and Technology, IEEE, pp 91–98
Heer J, Viégas FB, Wattenberg M (2007) Voyagers and voyeurs: supporting asynchronous collaborative information visualization. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 1029–1038
Hu T, Zheng H, Liang C, Zhu S, Imirzian N, Zhang Y, Wang C, Hughes DP, Chen DZ (2020) Antvis: A web-based visual analytics tool for exploring ant movement data. Vis Inf 4(1):58–70
Isenberg T, Isenberg P, Chen J, Sedlmair M, Möller T (2013) A systematic review on the practice of evaluating visualization. IEEE Trans Vis Comput Graph 19(12):2818–2827
Jung D, Kim W, Song H, Hwang Ji, Lee B, Kim B, Seo J (2017) Chartsense: Interactive data extraction from chart images. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 6706–6717
Kim NW, Schweickart E, Liu Z, Dontcheva M, Li W, Popovic J, Pfister H (2017a) Data-driven guides: supporting expressive design for information graphics. IEEE Trans Vis Comput Graph 23(1):491–500
Kim NW, Bach B, Im H, Schriber S, Gross M, Pfister H (2018) Visualizing nonlinear narratives with story curves. IEEE Trans Vis Comput Graph 24(1):595–604
Kim NW, Im H, Riche NH, Wang A, Gajos KZ, Pfister H (2019a) Dataselfie: Empowering people to design personalized visuals to represent their data. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems
Kim NW, Riche NH, Bach B, Xu G, Brehmer M, Hinckley K, Pahud M, Xia H, McGuffin MJ, Pfister H (2019b) Datatoon: Drawing data comics about dynamic networks with pen + touch interaction. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems
Kim YS, Reinecke K, Hullman J (2017b) Explaining the gap: Visualizing one’s predictions improves recall and comprehension of data. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 1375–1386
Klein G, Hoffman RR (2008) Macrocognition, mental models, and cognitive task analysis methodology. Naturalistic Decision Making and Macrocognition pp 57–80
Koytek P, Perin C, Vermeulen J, André E, Carpendale S (2018) Mybrush: brushing and linking with personal agency. IEEE Trans Vis Comput Graph 24(1):605–615
Lam H (2008) A framework of interaction costs in information visualization. IEEE Trans Vis Comput Graph 14(6):1149–1156
Lam H, Bertini E, Isenberg P, Plaisant C, Carpendale S (2011) Seven guiding scenarios for information visualization evaluation
Lee B, Kazi RH, Smith G (2013) Sketchstory: telling more engaging stories with data through freeform sketching conference on human factors in computing systems. IEEE Trans Vis Comput Graph 19(12):2416–2425
Lee B, Isaacs K, Szafir DA, Marai GE, Turkay C, Tory M, Carpendale S, Endert A (2019) Broadening intellectual diversity in visualization research papers. IEEE Comput Graph Appl 39(4):78–85
Li D, Mei H, Shen Y, Su S, Zhang W, Wang J, Zu M, Chen W (2018) Echarts: a declarative framework for rapid construction of web-based visualization. Vis Inf 2(2):136–146
Liu Z, Stasko J (2010) Mental models, visual reasoning and interaction in information visualization: A top-down perspective. IEEE Trans Vis Comput Graph 16(6):999–1008
Liu Z, Thompson J, Wilson A, Dontcheva M, Delorey J, Grigg S, Kerr B, Stasko J (2018) Data illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, p 123
Lu Z, Fan M, Wang Y, Zhao J, Annett M, Wigdor D (2019) Inkplanner: supporting prewriting via intelligent visual diagramming. IEEE Trans Vis Comput Graph 25(1):277–287
Mackinlay J, Hanrahan P, Stolte C (2007) Show me: automatic presentation for visual analysis. IEEE Trans Vis Comput Graph 13(6):1137–1144
Mei H, Ma Y, Wei Y, Chen W (2017) The design space of construction tools for information visualization: A survey. J Vis Lang Comput 44:120–132
Mei H, Chen W, Ma Y, Guan H, Hu W (2018) Viscomposer: a visual programmable composition environment for information visualization. Vis Inf 2(1):71–81
Mei H, Guan H, Xin C, Wen X, Chen W (2020) Datav: data visualization on large high-resolution displays. Vis Inf 4(3):12–23
Méndez GG, Hinrichs U, Nacenta MA (2017) Bottom-up vs. top-down: Trade-offs in efficiency, understanding, freedom and creativity with infovis tools. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 841–852
Méndez GG, Nacenta MA, Hinrichs U (2018) Considering agency and data granularity in the design of visualization tools. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems
Nacenta MA, Méndez GG (2017) ivolver: A visual language for constructing visualizations from in-the-wild data. In: Proceedings of ACM International Conference on Interactive Surfaces and Spaces, pp 438–441
Norman D (2013) The design of everyday things: Revised and expanded edition. Basic books
Pan J, Chen W, Zhao X, Zhou S, Zeng W, Zhu M, Chen J, Fu S, Wu Y (to appear) Exemplar-based layout fine-tuning for node-link diagrams. IEEE Transactions on Visualization and Computer Graphics. In Proceedings of IEEE VIS 2020
Pantazos K, Lauesen S (2012) Constructing visualizations with infovis tools-an evaluation from a user perspective. GRAPP/IVAPP 2012:731–736
Patterson RE, Blaha LM, Grinstein GG, Liggett KK, Kaveney DE, Sheldon KC, Havig PR, Moore JA (2014) A human cognition framework for information visualization. Comput Graph 42:42–58
Pike WA, Stasko J, Chang R, O’connell TA (2009) The science of interaction. Inf Vis 8(4):263–274
Ren D, Höllerer T, Yuan X (2014) ivisdesigner: expressive interactive design of information visualizations. IEEE Trans Vis Comput Graph 20(12):2092–2101
Ren D, Lee B, Brehmer M (2019) Charticulator: interactive construction of bespoke chart layouts. IEEE Trans Vis Comput Graph 25(1):789–799
Roberts JC, Ritsos PD, Jackson JR, Headleand C (2017) The explanatory visualization framework: an active learning framework for teaching creative computing using explanatory visualizations. IEEE Trans Vis Comput Graph 24(1):791–801
Romat H, Appert C, Pietriga E (2019a) Expressive authoring of node-link diagrams with graphies. IEEE Transactions on Visualization and Computer Graphics
Romat H, Henry Riche N, Hinckley K, Lee B, Appert C, Pietriga E, Collins C (2019b) Activeink: (th) inking with data. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 1–13
Saket B, Kim H, Brown ET, Endert A (2017) Visualization by demonstration: an interaction paradigm for visual data exploration. IEEE Trans Vis Comput Graph 23(1):331–340
Sarracino J, Barrios-Arciga O, Zhu J, Marcus N, Lerner S, Wiedermann B (2017) User-guided synthesis of interactive diagrams. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 195–207
Satyanarayan A, Heer J (2014a) Authoring narrative visualizations with ellipsis. Comput Graph Forum 33(3):361–370
Satyanarayan A, Heer J (2014b) Lyra: an interactive visualization design environment. Comput Graph Forum 33(3):351–360
Satyanarayan A, Russell R, Hoffswell J, Heer J (2016) Reactive vega: a streaming dataflow architecture conference on human factors in computing systems for declarative interactive visualization. IEEE Trans Vis Comput Graph 22(1):659–668
Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J (2017) Vega-lite: a grammar of interactive graphics. IEEE Trans Vis Comput Graph 23(1):341–350
Satyanarayan A, Lee B, Ren D, Heer J, Stasko J, Thompson J, Brehmer M, Liu Z (2019) Critical reflections on visualization authoring systems. IEEE Trans Vis Comput Graph 26(1):461–471
Scaife M, Rogers Y (1996) External cognition: how do graphical representations work? Int J Hum Comput Stud 45(2):185–213
Shen E, Li S, Cai X, Zeng L, Wang W (2014) Sketch-based interactive visualization: a survey. J Vis 17(4):275–294
Shneiderman B (1996) The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings 1996 IEEE Symposium on Visual Languages, IEEE, pp 336–343
Shu X, Wu J, Wu X, Liang H, Cui W, Wu Y, Qu H (2020) Dancingwords: exploring animated word clouds to tell stories. Journal of Visualization
Shu X, Wu A, Tang J, Bach B, Wu Y, Qu H (to appear) What makes a data-gif understandable? IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE VIS 2020)
Sicat R, Li J, Choi J, Cordeil M, Jeong WK, Bach B, Pfister H (2019) Dxr: a toolkit for building immersive data visualizations. IEEE Trans Vis Comput Graph 25(1):715–725
Tang J, Yu L, Tang T, Shu X, Ying L, Zhou Y, Ren P, Wu Y (to appeara) Narrative transitions in data videos. In: Proceedings of IEEE VIS 2020 Short Papers, IEEE
Tang T, Rubab S, Lai J, Cui W, Yu L, Wu Y (2019) istoryline: effective convergence to hand-drawn storylines. IEEE Trans Vis Comput Graph 25(1):769–778
Tang T, Li R, Wu X, Liu S, Knittel J, Koch S, Ertl T, Yu L, Ren P, Wu Y (to appearb) Plotthread: Creating expressive storyline visualizations using reinforcement learning. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE VIS 2020)
Tong C, Roberts R, Borgo R, Walton S, Laramee RS, Wegba K, Lu A, Wang Y, Qu H, Luo Q et al (2018) Storytelling and visualization: an extended survey. Information 9(3):65
Tory M, Moller T (2004) Human factors in visualization research. IEEE Trans Vis Comput Graph 10(1):72–84
Tversky B, Agrawala M, Heiser J, Lee P, Hanrahan P, Phan D, Stolte C, Daniel MP (2006) Cognitive design principles for automated generation of visualizations. From Research to Cognitive Technology, Applied Spatial Cognition, pp 53–75
Viegas FB, Wattenberg M, Van Ham F, Kriss J, McKeon M (2007) Manyeyes: a site for visualization at internet scale. IEEE Transactions on Visualization and Computer Graphics 13(6):
Wang Q, Li Z, Fu S, Cui W, Qu H (2019) Narvis: authoring narrative slideshows for introducing data visualization designs. IEEE Trans Vis Comput Graph 25(1):779–788
Wang Y, Chu X, Bao C, Zhu L, Deussen O, Chen B, Sedlmair M (2018a) Edwordle: consistency-preserving word cloud editing. IEEE Trans Vis Comput Graph 24(1):647–656
Wang Y, Zhang H, Huang H, Chen X, Yin Q, Hou Z, Zhang D, Luo Q, Qu H (2018b) Infonice: Easy creation of information graphics. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems
Weng D, Zheng C, Deng Z, Ma M, Bao J, Zheng Y, Xu M, Wu Y (to appear) Towards better bus networks: A visual analytics approach. IEEE Transactions on Visualization and Computer Graphics. In Proceedings of IEEE VIS 2020
Willett W, Heer J, Hellerstein J, Agrawala M (2011) Commentspace: structured support for collaborative visual analysis. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 3131–3140
Wongsuphasawat K, Moritz D, Anand A, Mackinlay J, Howe B, Heer J (2015) Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans Vis Comput Graph 22(1):649–658
Wongsuphasawat K, Qu Z, Moritz D, Chang R, Ouk F, Anand A, Mackinlay J, Howe B, Heer J (2017) Voyager 2: Augmenting visual analysis with partial view specifications. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 2648–2659
Xia H, Araujo B, Grossman T, Wigdor D (2016) Object-oriented drawing. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems, pp 4610–4621
Xia H, Henry Riche N, Chevalier F, De Araujo B, Wigdor D (2018) Dataink: Direct and creative data-oriented drawing. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems
Yalçin MA, Elmqvist N, Bederson BB (2016) Cognitive stages in visual data exploration. In: Proceedings of Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, pp 86–95
Yalcin MA, Elmqvist N, Bederson BB (2018) Keshif: rapid and expressive tabular data exploration for novices. IEEE Trans Vis Comput Graph 24(8):2339–2352
Ye S, Chen Z, Chu X, Wang Y, Fu S, Shen L, Zhou K, Wu Y (to appear) Shuttlespace: Exploring and analyzing movement trajectory in immersive visualization. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE VIS 2020)
Yi JS, ah Kang Stasko J, Y, Stasko J (2007) Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans Vis Comput Graph 13(6):1224–1231
Yoghourdjian V, Archambault D, Diehl S, Dwyer T, Klein K, Purchase HC, Wu HY (2018) Exploring the limits of complexity: a survey of empirical studies on graph visualisation. Vis Inf 2(4):264–282
Zacks JM, Tversky B (2003) Structuring information interfaces for procedural learning. J Exp Psychol Appl 9(2):88
Zgraggen E, Zeleznik R, Drucker SM (2014) Panoramicdata: data analysis through pen and touch. IEEE Trans Vis Comput Graph 20(12):2112–2121
Zhao Z, Marr R, Elmqvist N (2015) Data comics: Sequential art for data-driven storytelling. Tech Report
Zhu S, Sun G, Jiang Q, Zha M, Liang R (2020) A survey on automatic infographics and visualization recommendations. Vis Inf 4(3):24–40
Acknowledgements
The work was supported by NSFC (61761136020), NSFC-Zhejiang Joint Fund for the Integration of Industrialization and Informatization (U1609217), Zhejiang Provincial Natural Science Foundation (LR18F020001) and the 100 Talents Program of Zhejiang University. This project was also partially funded by Microsoft Research Asia.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rubab, S., Tang, J. & Wu, Y. Examining interaction techniques in data visualization authoring tools from the perspective of goals and human cognition: a survey. J Vis 24, 397–418 (2021). https://doi.org/10.1007/s12650-020-00705-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12650-020-00705-3