Sharon Oviatt
Jionghao Lin
Skip Abstract Section
Abstract
This research investigates whether students’ level of domain expertise can be detected during authentic learning activities by analyzing their physical activity patterns. More expert students reduced their manual activity by a substantial 50%, which was evident in fine-grained signal analyses and total rate of gesturing. The quality of experts’ discrete hand movements also averaged shorter in distance, briefer in duration, and slower in velocity than those of non-experts. Interestingly, experts adapted by nearly eliminating gestures on easier problems, while selectively increasing them on harder ones. They also strategically produced 62% more iconic gestures, which serve to retain spatial information in working memory while extracting inferences required to solve problems correctly. These findings highlight the close relation between hand movements and mental state and, more specifically, that hand movements provide an unusually clear window on students’ level of domain expertise. Embodied Cognition and Limited Resource theories only partially account for the present findings, which specify future directions for theoretical work.
- M. W. Alibali. 2005. Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spat. Cogn. Comput. 5, 4 (2005), 307–331.Google Scholar
Cross Ref
- M. W. Alibali and M. J. Nathan. 2012. Embodiment in mathematics teaching and learning: Evidence from learners' and teachers' gestures. J. Learn. Sci. 21, 2 (2012), 247–286.Google ScholarCross Ref
- A. M. Arthur, R. Lunsford, M. Wesson, and S. Oviatt. 2006. Prototyping novel collaborative multimodal systems: Simulation, data collection and analysis tools for the next decade. In Proceedings of the 8th International Conference on Multimodal Interfaces. 209–216. Google ScholarDigital Library
- K. Atit, K. Gagnier, and T. F. Shipley. 2015. Student gestures aid penetrative thinking. J. Geosci. Educ. 63, 1 (2015), 66–72.Google ScholarCross Ref
- A. Baddeley. 1986. Working Memory. Oxford University Press, New York, NY.Google Scholar
- L. W. Barsalou. 1999. Perceptual symbol systems. Behav. Brain Sci. 22, 4 (1999), 577–660.Google ScholarCross Ref
- J. Black, K. Isaacs, B. Anderson, A. Alcantara, and W. Greenough. 1990. Learning causes synaptogenesis, whereas motor activity causes angiogenesis in cerebellar cortex of adult rats. Proc. Natl. Acad. Sci. U.S.A. 87, 14 (1990), 5568–72.Google ScholarCross Ref
- N. Brooks, D. Barner, M. Frank, and S. Goldin-Meadow. 2018. The role of gesture in supporting mental representations: The case of mental abacus arithmetic. Cogn. Sci. 42, 2 (2018), 554–575.Google ScholarCross Ref
- Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh. 2018. OpenPose: Real-time multi-person 2D pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7291–7299.Google Scholar
- W. Chase and K. Ericsson. 1982. Skill and working memory. In The Psychology of Learning and Motivation, G. Bower (Ed.). Academic Press, New York, 1–58.Google Scholar
- W. Chase and H. Simon. 1973. The mind's eye in chess. In Visual Information Processing, W. G. Chase (Ed.). Academic Press, San Diego, CA, 215–81.Google Scholar
- M. Chu and S. Kita. 2008. Spontaneous gestures during mental rotation tasks: Insights into the microdevelopment of the motor strategy. J. Exp. Psychol.: General 137 (2008), 706–723.Google ScholarCross Ref
- M. Chu and S. Kita. 2011. The nature of gestures' beneficial role in spatial problem solving. J. Exp. Psychol.: General 140, 1 (2011), 102–116.Google ScholarCross Ref
- A. Clark. 2013. Gesture as thought? In The Hand, An Organ of the Mind, Z. Radman (Ed.). MIT Press, Cambridge MA, 255–268.Google Scholar
- H. Clark and S. Brennan. 1991. Grounding in communication. In Perspectives on Socially Shared Cognition L. Resnick, B. Levine, M. John, and S. D. Teasley (Eds.). American Psychological Association, 127–149.Google Scholar
- H. H. Clark and E. F. Schaefer. 1989. Contributing to discourse. Cogn. Sci. 13, 2 (1989), 259–294.Google ScholarCross Ref
- R. A. Clark, B. F. Mentiplay, E. Hough, and Y. H. Pua. 2019. Three-dimensional cameras and skeleton pose tracking for physical function assessment: A review of uses, validity, current developments and kinect alternatives. Gait Posture 68 (2019), 193–200.Google ScholarCross Ref
- A. Comblain. 1994. Working memory in down's syndrome: Training the rehearsal strategy. Down's Syndr. Res. Pract. 2, 3 (1994), 123–126.Google ScholarCross Ref
- S. W. Cook, T. K. Yip, and S. Goldin-Meadow. 2012. Gestures, but not meaningless movements, lighten working memory load when explaining math. Lang. Cogn. Process. 27, 4 (2012), 594–610.Google ScholarCross Ref
- J. P. De Ruiter. 1998. Gesture and Speech Production. Ph.D. Thesis. Radboud University Nijmegen Nijmegen, The Netherlands.Google ScholarCross Ref
- C. Eielts, W. Pouw, K. Ouwehand, T. Van Gog, R. A. Zwaan, and F. Paas. 2018. Co-thought gesturing supports more complex problem solving in subjects with lower visual working-memory capacity. Psychol. Res. (2018), 1–12.Google Scholar
- K. A. Ericsson, R. T. Krampe, and C. Tesch-Romer. 1993. The role of deliberate practice in the acquisition of expert performance. Psychol. Rev. 100, 3 (1993), 363–406.Google ScholarCross Ref
- P. Ferchmin and E. Bennett. 1975. Direct contact with enriched environment is required to alter cerebral weight in rats. J. Compar. Physiol. Psychol. 88 (1975), 360–67.Google ScholarCross Ref
- S. Goldin-Meadow and S. Beilock. 2010. Action's influence on thought: The case of gesture. Perspect. Psychol. Sci. 5, 6 (2010), 664–674.Google ScholarCross Ref
- S. Goldin-Meadow, S. C. Levine, E. Zinchenko, T. K. Yip, N. Hemani, and L. Factor. 2012. Doing gesture promotes learning a mental transformation task better than seeing gesture. Dev. Sci. 15, 6 (2012), 876–884.Google ScholarCross Ref
- S. Goldin-Meadow, H. Nusbaum, S. D. Kelly, and S. Wagner. 2001. Explaining math: Gesturing lightens the load. Psychol. Sci. 12, 6 (2001), 516–522.Google ScholarCross Ref
- N. M. Hill and W. Schneider. 2006. Brain changes in the development of expertise: Neuroanatomical and neurophysiological evidence about skill-based adaptations. In The Cambridge Handbook of Expertise and Expert Performance. Cambridge University Press, UK, 653–682.Google Scholar
- A. Holtmaat, L. Wilbrecht, G. Knott, E. Welker, and K. Svoboda. 2006. Experience-dependent and cell-type-specific spine growth in the neocortex. Nature 441 (2006), 979–78.Google ScholarCross Ref
- A. B. Hostetter and M. W. Alibali. 2008. Visible embodiment: Gestures as simulated action. Psychonom. Bull. Rev. 15, 3 (2008), 495–514.Google ScholarCross Ref
- A. Hostetter and M. Alibali. 2019. Gesture as simulated action: Revisitng the framework. Psychonom. Bull. Rev. 26 (2019), 721–752.Google ScholarCross Ref
- M. Howison, D. Trninic, D. Reinholz, and D. Abrahamson. 2011. The mathematical imagery trainer: From embodied interaction to conceptual learning. In Proceedings of the CHI Conference, ACM Press, New York, NY, 1989–1998. Google ScholarDigital Library
- K. H. James. 2010. Sensori-motor experience leads to changes in visual processing in the developing brain. Dev. Sci. 13, 2 (2010), 279–288.Google ScholarCross Ref
- K. James and S. Swain. 2010. Only self-generated actions create sensori-motor systems in the developing brain. Dev. Sci. (2010), 1–6.Google Scholar
- A. Kelly and H. Garavan. 2005. Human functional neuroimaging of brain changes associated with practice. Cerebr. Cortex 15, 8 (2005), 1089–1102.Google ScholarCross Ref
- R. Kirk. 2013. Experimental Design: Procedures for the Behavioral Sciences (4th ed.), Sage, Thousand Oaks, CA.Google Scholar
- S. Kita, M. W. Alibali, and M. Chu. 2017. How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychol. Rev. 124, 3 (2017), 245.Google ScholarCross Ref
- J. Kleim, K. Vij, J. Kelly, D. Ballard, and W. Greenough. 1997. Learning-dependent synaptic modifications in the cerebellar cortex of the adult rat persist for at least 4 weeks. J. Neurosci. 17 (1997), 717–721.Google ScholarCross Ref
- B. Lindblom. 1990. Explaining phonetic variation: A sketch of the H&H theory. In Speech Production and Speech Modelling. Springer, Netherlands, 403–439.Google Scholar
- M. Longcamp, C. Boucard, J.-C. Gilhodes, J.-L. Anton, M. Roth, B. Nazarian, and J.-L. Velay. 2008. Learning through hand- or typewriting influences visual recognition of new graphic shapes: Behavioral and functional imaging evidence. J. Cogn. Neurosci. 20, 5 (2008), 802–815. Google ScholarDigital Library
- A. R. Luria. 1961. The Role of Speech in the Regulation of Normal and Abnormal Behavior. Liveright, Oxford, UK.Google Scholar
- J. Markham and W. Greenough. 2004. Experience-driven brain plasticity: Beyond the synapse. Neuron Glia Biol. 1, 4 (2004), 351–363.Google ScholarCross Ref
- D. McNeill. 2005. Gesture and Thought. University of Chicago Press, Chicago.Google Scholar
- G. Miller, E. Galanter, and K. Pribram. 1960. Plans and the Structure of Behavior. Holt, Rinehart & Winston, New York, NY.Google Scholar
- S. L. Oviatt. 2013a. The Design of Future Educational Interfaces. Routledge Press.Google Scholar
- S. Oviatt. 2013. Problem solving, domain expertise and learning: Ground-truth performance results for math data corpus. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction. 569–574. Google ScholarDigital Library
- S. Oviatt. 2017. Theoretical foundations of multimodal interfaces and systems. In The Handbook of Multimodal-Multisensor Interfaces, Volume I: Foundations, User Modeling, and Common Modality Combinations, O. Sharon, S. Björn, C. Philip, S. Daniel, and P. Gerasimos (eds.). Morgan & Claypool, New York, NY, 19–50. Google ScholarDigital Library
- S. Oviatt. 2018. Ten opportunities and challenges for advancing student-centered multimodal learning analytics. In Proceedings of the International Conference on Multimodal Interaction. 87–94. Google ScholarDigital Library
- S. Oviatt, A. Arthur, Y. Brock, and J. Cohen. 2007. Expressive pen-based interfaces for math education. In Proceedings of the 8th International Conference on Computer Supported Collaborative Learning. 573–582. Google ScholarDigital Library
- S. Oviatt, A. Cohen, N. Weibel, K. Hang, and K. Thompson. 2014. Multimodal learning analytics data resources: Description of math data corpus and coded documents. In Proceedings of the 3rd International Data-Driven Grand Challenge Workshop on Multimodal Learning Analytics. 414.Google Scholar
- S. Oviatt, R. Coulston, and R. Lunsford. 2004. When do we interact multimodally?: Cognitive load and multimodal communication patterns. In Proceedings of the 6th International Conference on Multimodal Interfaces. 129–136. Google ScholarDigital Library
- S. Oviatt, J. Grafsgaard, L. Chen, and X. Ochoa. 2018. Multimodal learning analytics: Assessing learners' mental state during the process of learning. In The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Cognition and Emotion. Morgan & Claypool, New York, NY, 331–374. Google ScholarDigital Library
- S. Oviatt, K. Hang, J. Zhou, K. Yu, and F. Chen. 2018. Dynamic handwriting signal features predict domain expertise. ACM Trans. Interact. Intell. Syst. 8, 3 (2018), 1–21. Google ScholarDigital Library
- S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, and G. Potamianos. 2017. The Handbook of Multimodal-Multisensor Interfaces, Volume 1: Foundations, User Modeling, and Common Modality Combinations. Morgan & Claypool, New York, NY. Google ScholarDigital Library
- F. Phillips, M. Natter, and E. Egan. 2015. Magically deceptive biological motion-The French drop sleight, Front. Psychol. 6, (371) 1–10.Google Scholar
- R. M. Ping and S. Goldin-Meadow. 2008. Hands in the air: Using ungrounded iconic gestures to teach children conservation of quantity. Dev. Psychol. 44, 5 (2008), 1277–1287.Google ScholarCross Ref
- G. Rizzolatti, L. Fogassi, and V. Gallese. 2001. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat. Rev. Neurosci. 2, 9 (2001), 661.Google ScholarCross Ref
- L. Rueckert, R. B. Church, A. Avila, and T. Trejo. 2017. Gesture enhances learning of a complex statistical concept. Cogn. Res.: Principl. Implicat. 2, 1 (2017), 2.Google ScholarCross Ref
- A. Sale, N. Berardi, and L. Maffei. 2009. Enrich the environment to empower the brain. Trends Neurosci. 32, 233–9.Google ScholarCross Ref
- R. Shiffrin and W. Schneider. 1977. Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychol. Rev. 84 (1977), 127–190.Google ScholarCross Ref
- A. Sriramulu, J. Lin, and S. Oviatt. 2019. Dynamic adaptive gesturing as a predictor of domain expertise in mathematics. In Proceedings of the International Conference on Multimodal Interfaces. ACM, New York, NY. Google ScholarDigital Library
- G. J. Trafton, S. B. Trickett, C. A. Stitzlein, L. Saner, C. D. Schunn, and S. S. Kirschenbaum. 2006. The relationship between spatial transformations and iconic gestures. Spatial Cogn. Comput. 6, 1 (2006), 1–29.Google ScholarCross Ref
- B. Tversky. 2009. Spatial cognition: Embodied and situated. In The Cambridge Handbook of Situated Cognition, P. Robbins and M. Aydede (Eds.). Cambridge University Press, Cambridge, UK, 117–133.Google Scholar
- F. Varela, E. Thompson, and E. Rosch. 1992. The Embodied Mind: Cognitive Science and Human Experience, MIT Press, Cambridge, MA.Google Scholar
- L. Vygotsky. 1962. Thought and Language. MIT Press, Cambridge MA. (Translated by E. Hanfmann and G. Vakar from 1934 original).Google Scholar
- L. Vygotsky. 1987. The Collected Works of L. S. Vygotsky, Volume I: Problems of General Psychology, Edited and translated by N. Minick. Plenum, New York, NY.Google Scholar
- M. Wilson. 2002. Six views of embodied cognition. Psychonom. Bull. Rev. 9, 4 (2002), 625–636.Google ScholarCross Ref
- G. Yang, F. Pan, and W. B. Gan. 2009. Stably maintained dendritic spines are associated with lifelong memories. Nature 462 (2009), 920–924.Google ScholarCross Ref
- J. Zhou, K. Hang, S. Oviatt, K. Yu, and F. Chen. 2014. Combining empirical and machine learning techniques to predict math expertise using pen signal features. In Proceedings of the ACM Data-Driven Grand Challenge Workshop on Multimodal Learning Analytics. 29–36. Google ScholarDigital Library
Index Terms
- I Know What You Know: What Hand Movements Reveal about Domain Expertise
Recommendations
Dynamic Adaptive Gesturing Predicts Domain Expertise in Mathematics
ICMI '19: 2019 International Conference on Multimodal InteractionEmbodied Cognition theorists believe that mathematics thinking is embodied in physical activity, like gesturing while explaining math solutions. This research asks the question whether expertise in mathematics can be detected by analyzing students’ rate ...
Problem solving, domain expertise and learning: ground-truth performance results for math data corpus
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interactionProblem solving, domain expertise, and learning are analyzed for the Math Data Corpus, which involves multimodal data on collaborating student groups as they solve math problems together across sessions. Compared with non-expert students, domain experts ...
User Elicitation on Single-hand Microgestures
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsGestural interaction has become increasingly popular, as enabling technologies continue to transition from research to retail. The mobility of miniaturized (and invisible) technologies introduces new uses for gesture recognition. This paper investigates ...
Comments