Abstract
As commercial pen-centric systems proliferate, they create a parallel need for analytic techniques based on dynamic writing. Within educational applications, recent empirical research has shown that signal-level features of students’ writing, such as stroke distance, pressure and duration, are adapted to conserve total energy expenditure as they consolidate expertise in a domain. The present research examined how accurately three different machine-learning algorithms could automatically classify users’ domain expertise based on signal features of their writing, without any content analysis. Compared with an unguided machine-learning classification accuracy of 71%, hybrid methods using empirical-statistical guidance correctly classified 79–92% of students by their domain expertise level. In addition to improved accuracy, the hybrid approach contributed a causal understanding of prediction success and generalization to new data. These novel findings open up opportunities to design new automated learning analytic systems and student-adaptive educational technologies for the rapidly expanding sector of commercial pen systems.
- E. Andre. In press. Real-time sensing of affect and social signals in a multimodal context. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
- Apple. 2017. Retrieved January 17, 2017 from http://www.apple.com/apple-pencil/.Google Scholar
- A. Arthur, R. Lunsford, M. Wesson, and S. Oviatt. 2006. Prototyping novel collaborative multimodal systems: Simulation, data collection and analysis tools for the next decade. In Proceedings of the 8th ACM International Conference on Multimodal Interaction. ACM, New York, NY, 209--226. Google ScholarDigital Library
- R. Baker and K. Yacef. 2009. The state of educational data mining in 2009: A review and future visions. J. Educ. Data Min. 1 (2009), 3--17 .Google Scholar
- R. Baker and G. Siemens. 2014. Learning analytics and educational data mining. In Cambridge Handbook of the Leaning Sciences (2nd ed), R. K. Sawyer (Ed.). Cambridge University Press, New York, NY, 253--272.Google Scholar
- L. Bourne, J. Kole, and A. Healy. 2014. Expertise: Defined, described and explained. Front. Psychol. 5, 186 (2014), 1--4.Google ScholarCross Ref
- M. Burzo, M. Abouelenien, V. Perez-rosas, and R. Mihalcea. In press. Multimodal deception detection. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
- A. Caramazza and A. H. Hillis. 1991. Lexical organization of nouns and verbs in the brain. Nature 349 (1991), 788--790.Google ScholarCross Ref
- P. C. H. Cheng and H. Rojas-anaya. 2007. Measuring mathematical formula writing competence: An application of graphical protocol analysis. In Proceedings of the 29th Conference of the Cognitive Science Society. D. S. McNamara and J. G. Trafton (Eds.). Cognitive Science Society, Austin, TX, 869--874.Google Scholar
- P. C. H. Cheng and H. Rojas-anaya. 2008. A graphical chunk production model: Evaluation using graphical protocol analysis with artificial sentences. In Proceedings of the 30th Annual Conference of the Cognitive Science Society. B. C. Love, K. McRae, and V. M. Sloutsky (Eds). Cognitive Science Society, Austin, TX, 1972--1977.Google Scholar
- J. Cohn, N. Cummins, J. Epps, R. Goecke, J. Joshi, and S. Scherer. In press. Multimodal assessment of depression and related disorders based on behavioural signals. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
- S. D'mello, N. Bosch, and H. Chen. In press. Multimodal-multisensor affect detection. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
- P. Domingos. 2012. A few useful things to know about machine learning. Commun. ACM 55, 10 (2012), 78--87. Google ScholarDigital Library
- K. A. Ericsson, R. T. Krampe, and C. Tesch-romer. 1993. The role of deliberate practice in the acquisition of expert performance. Psychol. Rev. 100, 3 (1993), 363--406.Google ScholarCross Ref
- B. Jee, D. Gentner, K. Forbus, B. Sageman, and D. Uttal. 2009. Drawing on experience: Use of sketching to evaluate knowledge of spatial scientific concepts. In Proceedings of the 31st Conference of Cognitive Science Society. Cognitive Science Society, Amsterdam.Google Scholar
- B. Jee, D. Gentner, D. Uttal, B. Sageman, K. Forbus, C. Manduca, C. Ormand, T. Shipley, and B. Tikoff. 2014. Drawing on experience: How domain knowledge is reflected in sketches of scientific structures and processes. In Research in Science Education. Springer, Dordrecht.Google Scholar
- D. Kahnemann. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, NY.Google Scholar
- J. M. Kantner and K. Veermanachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In Proceedings of the IEEE/ACM Data Science and Advance Analytics Conference.Google Scholar
- M. Lewis. 2014. Flash Boys. W.W. Norton and Company, New York, NY.Google Scholar
- Livescribe. 2017. Retrieved January 17, 2017 from http://www.livescribe.com/en-us/smartpen/ls3/.Google Scholar
- L. Martin and D. Schwartz. 2009. Prospective adaptation in the use of external representations. Cogn. Instruct. 27, 4 (2009), 370--400.Google ScholarCross Ref
- Microsoft. 2016. Retrieved April 1, 2016 from https://www.microsoft.com/surface/enus?8SEMID=18WT.srch=18ocid=_SEM_GOO_MSBranded_INV_enUS_+microsoft%20+surface%20+tablets8wt.mc_id=_SEM_GOO_MSBranded_INV_en-US_+microsoft%20+surface%20+tablets.Google Scholar
- MyScript. 2017. Retrieved January 17, 2017, http://myscript.com/technology/.Google Scholar
- Neo Smartpen. 2017; http://www.neosmartpen.com (retrieved Jan. 17, 2017).Google Scholar
- X. Ochoa, K. Chiluiza, G. Mendez, G. Luzardo, B. Guaman, and J. Castells. 2013. Expertise estimation based on simple multimodal features. In Proceedings of the 15th ACM International Conference on Multimodal Interaction. ACM Press, New York, NY. 583--590. Google ScholarDigital Library
- Orange. 2013. Retrieved October 31, 2013 from http://orange.biolab.si/.Google Scholar
- S. L Oviatt. 1996. User-centered design of spoken language and multimodal interfaces. IEEE Multimedia 3, 4 (1996), 26--35. Google ScholarDigital Library
- S. L. Oviatt. 2012. Multimodal interfaces. In The Human--Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications (3rd ed.), J. Jacko (Ed.) Lawrence Erlbaum and Associates, Mahwah, NJ. 405--430.Google Scholar
- S. L. Oviatt. 2013a. The Design of Future Educational Interfaces. Routledge Press, New York, NY.Google Scholar
- S. L. Oviatt. 2013b. Problem solving, domain expertise and learning: Ground-truth performance results for math data corpus. In Proceedings of the 2nd International Grand Challenge Workshop on Multimodal Learning Analytics. ACM Press, New York, NY. Google ScholarDigital Library
- S. L. Oviatt, A. Arthur, Y. Brock, and J. Cohen. 2007. Expressive pen-based interfaces for math education. In Proceedings of Computer-Supported Collaborative Learning Conference. ISLS Chinn, Erkens and Puntambekar (Eds.) 8, 2, 569--578. Google ScholarDigital Library
- S. L. Oviatt, A. Arthur and J. Cohen. 2006. Quiet interfaces that help students think. In Proceedings of the ACM Conference on User Interface Software Technology. ACM Press, New York, NY. 191--200. Google ScholarDigital Library
- S. L. Oviatt and A. Cohen. 2010. Toward high-performance communication interfaces for science problem solving. J. Sci. Educ. Technol. 19, 6 (2010), 515--531.Google ScholarCross Ref
- S. L. Oviatt and A. Cohen. 2013. Written and multimodal representations as predictors of expertise and problem-solving success in mathematics. In Proceedings of the 15th ACM International Conference on Multimodal Interaction. ACM Press, New York, NY. 599--606. Google ScholarDigital Library
- S. L. Oviatt and A. Cohen. 2014. Written activity, representations, and fluency as predictors of domain expertise in mathematics. In Proceedings of the 16th ACM International Conference on Multimodal Interfaces. ACM Press, New York, NY. Google ScholarDigital Library
- S. L. Oviatt and P. R. Cohen. 2015. The Paradigm Shift to Multimodality in Contemporary Computer Interfaces. Human-Centered Interfaces Synthesis series, Jack Carroll (Ed.). Morgan Claypool, San Rafael, CA. Google ScholarDigital Library
- S. L. Oviatt, A. Cohen, A. Miller, K. Hodge, and A. Mann. 2012. The impact of interface affordances on human ideation, problem solving and inferential reasoning. ACM Transactions on Computer Human Interaction. ACM Press, New York, NY. 19, 3 (2012), 1--30. Google ScholarDigital Library
- S. L. Oviatt, A. Cohen, and N. Weibel. 2013. Multimodal learning analytics: Description of math data corpus for ICMI grand challenge workshop. In Proceedings of the 2nd International Workshop on Multimodal Learning Analytics. ACM Press, New York, NY. Google ScholarDigital Library
- S. L. Oviatt, J. Graafsgard, L. Chen, and X. Ochoa. In press. Multimodal learning analytics: Assessing learners’ mental state during the process of learning. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
- S. L. Oviatt, K. Hang, and J. Zhou. Unpublished. Predicting expertise from adaptive energy expenditure during writing.Google Scholar
- S. L. Oviatt, K. Hang, J. Zhou, K. YU, and F. Chen. 2016. Dynamic handwriting signal features predict domain expertise. Incaa Designs Technical Report 52016. Incaa Designs, Deer Harbor, WA.Google Scholar
- Samsung. 2017. Retrieved January 17, 2017 from http://www.samsung.com/global/galaxy/galaxy-tab-pro-s/.Google Scholar
- B. Schuller. In press. Multimodal user state and trait recognition: An overview. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
- R. Shiffren and W. Schneider. 1977. Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychol. Rev. 84 (1977), 127--190.Google ScholarCross Ref
- N. Singer. 2014. With tech taking over schools, worries rise. New York Times, Sept. 15, 2014.Google Scholar
- Smart. 2016. Retrieved April 1, 2016 from http://smartkapp.com/Products/kapp.Google Scholar
- I. H. Witten, E. Frank, and M. A. Hall. 2011. Data Mining: Practical Machine Learning Tools and Techniques (3rd ed.). Morgan Kaufmann: Burlington, MA. Google ScholarDigital Library
- M. Worsley and P. Blikstein. 2010. Toward the development of learning analytics: Student speech as an automatic and natural form of assessment. In Proceedings of the American Educational Research Association Conference.Google Scholar
- K. Yu, J. Epps, and F. Chen. 2011. Cognitive load evaluation of handwriting using stroke-level features. In Proceedings of the 16th International Conference on Intelligent User Interfaces. ACM Press, New York, NY. 423--426. Google ScholarDigital Library
- K. Yu, J. Epps, and F. Chen. 2013. Mental workload classification via online writing features. In Proceedings of the 12th IEEE International Conference on Document Analysis and Recognition. 1110--1114. Google ScholarDigital Library
- J. Zhou, K. Hang, S. Oviatt, K. Yu, and F. Chen. 2014. Combining empirical and machine learning techniques to predict math expertise using pen signal features. In Proceedings of the 3rd International Conference on Multimodal Interaction's Grand Challenge Workshop on Multimodal Learning Analytics. ACM Press, New York, NY. Google ScholarDigital Library
- J. Zhou, J. Sun, F. Chen, Y. Wang, R. Taib, A. Khawaji, and Z. Li. 2015. Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Trans. Comput.-Hum. Interact. 21, 6 (2015), 1--33. Google ScholarDigital Library
- J. Zhou, K. Yu, F. Chen, Y. Wang, and S. Arshad. In press. Multimodal behavioral and physiological signals as indicators of cognitive load. The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition, S. Oviatt, B. Schuller, P. Cohen, D. Sonntag, G. Potamianos, and A. Krueger (Eds.). San Rafael, CA: Morgan and Claypool.Google Scholar
Index Terms
- Dynamic Handwriting Signal Features Predict Domain Expertise
Recommendations
Combining empirical and machine learning techniques to predict math expertise using pen signal features
MLA '14: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand ChallengeMultimodal learning analytics aims to automatically analyze students' natural communication patterns based on speech, writing, and other modalities during learning activities. This research used the Math Data Corpus, which contains time-synchronized ...
On-line handwriting character recognition method with directional features and direction-change features
ICDAR '97: Proceedings of the 4th International Conference on Document Analysis and RecognitionWe propose a new on-line recognition method to recognize handwritten cursive-style characters correctly. Our method simultaneously uses both directional features, otherwise known as off-line features, and direction-change features, which we designed as ...
Comments