Abstract
Graphs are often accompanied by text, i.e. linguistically coded information, augmenting the information presented diagrammatically. Thus, graph comprehension by humans often constitutes comprehension and integration of information provided by different representational modalities, namely graphical elements and verbal constituents. In this study we focus on textual annotations to line graphs providing information about events, processes and their temporal properties as well as temporal relations about the events and processes in question. We present results of an experimental investigation on parameters which influence subject’s interpretations concerning the temporal properties of the annotated events and on eye movement behavior. In particular, we discuss the role of graph shape and the role of graphical means for relating textual annotations and determined parts of the graph line.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bernsen, N.O.: Foundations of Multimodal Representations: A Taxonomy of Representational Modalities. Interacting with Computers 6, 347–371 (1994)
Hegarty, M.: The Mechanics of Comprehension and Comprehension of Mechanics. In: Rayner, K. (ed.) Eye Movements and Visual Cognition: Scene Perception and Reading, pp. 428–443. Springer, New York (1992)
Butcher, K.R.: Learning from Text with Diagrams: Promoting Mental Model Development and Inference Generation. Journal of Educational Psychology 98, 182–197 (2006)
Narayanan, N.H., Hegarty, M.: On Designing Comprehensible Interactive Hypermedia Manuals. International Journal of Human Computer Studies 48, 267–301 (1998)
Scaife, M., Rogers, Y.: External Cognition: How Do Graphical Representations Work? International Journal of Human Computer Studies 45, 185–213 (1996)
Peebles, D.J., Cheng, P.C.-H.: Extending Task Analytic Models of Graph-based Reasoning: A Cognitive Model of Problem Solving with Cartesian Graphs in ACT-R/PM. Cognitive Systems research 3, 77–86 (2002)
Pinker, S.: A Theory of Graph Comprehension. In: Freedle, R. (ed.) Artificial intelligence and the Future of Testing, pp. 73–126. Erlbaum, Hillsdale (1990)
Mautone, P.D., Mayer, R.E.: Cognitive Aids for Guiding Graph Comprehension. Journal of Educational Psychology 99, 640–652 (2007)
Kosslyn, S.M.: Image and Mind. Harvard University Press, Cambridge (1980)
Kosslyn, S.M.: Understanding Charts and Graphs. Applied Cognitive Psychology 3, 185–226 (1989)
Tversky, B.: Semantics, Syntax, and Pragmatics of Graphics. In: Holmqvist, K., Ericsson, Y. (eds.) Language and Visualisation, pp. 141–158. Lund University Press, Lund (2004)
Habel, C., Acarturk, C.: On Reciprocal Improvement in Multimodal Generation: Co-reference by Text and Information Graphics. In: van der Sluis, I., Theune, M., Reiter, E., Krahmer, E. (eds.) Workshop on Multimodal Output Generation (MOG 2007), Aberdeen, United Kingdom, pp. 69–80 (2007)
Acarturk, C., Habel, C., Cagiltay, K., Alacam, O.: Multimodal Comprehension of Language and Graphics: Graphs with and without Annotations (in press, accepted for publication in Journal of Eye Movement Research)
Underwood, G., Jebbett, L., Roberts, K.: Inspecting Pictures for Information to Verify a Sentence: Eye Movements in General Encoding and in Focused search. The Quarterly Journal of Experimental Psychology 57A, 165–182 (2004)
Just, M.A., Carpenter, P.A.: A Theory of Reading: From Eye Fixations to Comprehension. Psychological Review 87, 329–354 (1980)
Anderson, J.R., Bothell, D., Douglass, S.: Eye Movements Do Not Reflect Retrieval Processes: Limits of the Eye-mind Hypothesis. Psychological Science 15, 225–231 (2004)
Cohen, E., Singh, M.: Geometric Determinants of Shape Segmentation: Tests Using Segment Identification. Vision Research 47, 2825–2840 (2007)
De Winter, J., Wagemans, J.: Segmentation of Object Outlines into Parts: A Large-scale Integrative Study. Cognition 99, 275–325 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Acarturk, C., Habel, C., Cagiltay, K. (2008). Multimodal Comprehension of Graphics with Textual Annotations: The Role of Graphical Means Relating Annotations and Graph Lines. In: Stapleton, G., Howse, J., Lee, J. (eds) Diagrammatic Representation and Inference. Diagrams 2008. Lecture Notes in Computer Science(), vol 5223. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87730-1_30
Download citation
DOI: https://doi.org/10.1007/978-3-540-87730-1_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87729-5
Online ISBN: 978-3-540-87730-1
eBook Packages: Computer ScienceComputer Science (R0)