Skip to main content
SpringerLink
Log in

Automatic detection of users’ skill levels using high-frequency user interface events

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Computer users have different levels of system skills. Moreover, each user has different levels of skill across different applications and even in different portions of the same application. Additionally, users’ skill levels change dynamically as users gain more experience in a user interface. In order to adapt user interfaces to the different needs of user groups with different levels of skills, automatic methods of skill detection are required. In this paper, we present our experiments and methods, which are used to build automatic skill classifiers for desktop applications. Machine learning algorithms were used to build statistical predictive models of skill. Attribute values were extracted from high frequency user interface events, such as mouse motions and menu interactions, and were used as inputs to our models. We have built both task-independent and task-dependent classifiers with promising results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anderson J.R.: Acquisition of cognitive skill. Psychol. Rev. 89(4), 369–406 (1982)

    Article  Google Scholar 

  • Anderson J.R.: The Architecture of Cognition. Harvard University Press, Cambridge (1983)

    Google Scholar 

  • Barfield W.: Expert-novice differences for software: implications for problem-solving and knowledge acquisition. Behav. Inf. Technol. 5(1), 15–29 (1986)

    Article  Google Scholar 

  • Beale, R., Finlay, J., Austin, J., Harrison, M.: User modelling by classification: a neural-based approach. New Developments in Neural Computing, In: Taylor, J.G., Mannion, C.L.T. (eds.) (Adam Hilger), pp. 103–110 (1989)

  • Benyon D.: Adaptive systems: a solution to usability problems. User Model. User-adapt. Interact. 3(1), 65–87 (1993)

    Article  Google Scholar 

  • Brewer M.: Research design and issues of validity. In: Reis, H., Judd, C. (eds) Handbook of research methods in social and personality psychology, Cambridge University Press, Cambridge (2000)

    Google Scholar 

  • Browne, D., Totterdell, P., Norman, M. (eds): Adaptive User Interfaces. Academic Press Ltd, London (1990)

    Google Scholar 

  • Brusilovsky P.: Methods and techniques of adaptive hypermedia. User Model. User-adapt. Interact. 6(2–3), 87–129 (1996)

    Article  Google Scholar 

  • Card S.K., Moran T.P., Newell A.: The keystroke-level model for user performance time with interactive systems. Commun. ACM. 23(7), 396–410 (1980)

    Article  Google Scholar 

  • Caruana R.: Multitask learning. Mach. Learn. 28, 41–75 (1997)

    Article  Google Scholar 

  • Chase W.G., Simon H.A.: Perception in chess. Cogn. Psychol. 4, 55–81 (1973)

    Article  Google Scholar 

  • Cheung Chiu B., Webb G.I.: Using decision trees for agent modeling: improving prediction performance. User Model. User-adapt. Interact. 8(1–2), 131–152 (1998)

    Article  Google Scholar 

  • Chin, D.N.: KNOME: modeling what the user knows in UC. In: Kobsa A., Wahlster W. (eds.) User Models in Dialog Systems, pp. 74–107. Springer, Berlin (Symbolic computation series edited by Loveland D.W.) (1989)

  • Crossman E.R.F.W.: A theory of the acquisition of speed-skill. Ergonomics 2, 153–156 (1959)

    Article  Google Scholar 

  • Darzi A., Mackay S.: Skills assessment of surgeons. Surgery 131(2), 121–124 (2002)

    Article  Google Scholar 

  • Datta V., Mackay S., Mandalia M., Darzi A.: The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in laboratory-based model. J. Am. Coll. Surg. 193, 479–485 (2001)

    Article  Google Scholar 

  • Datta V., Mandalia M., Mackay S., Chang A., Cheshire N., Darzi A.: Relationship between skill and outcome in the laboratory-based model. Surgery 131(3), 318–323 (2001)

    Article  Google Scholar 

  • Desmarais, M.C., Liu, J.: Exploring the applications of user-expertise assessment for intelligent interfaces. In: Proceedings of the INTERACT ‘93 and CHI ‘93 Conference on Human Factors in Computing Systems, (pp. 308–313). Amsterdam, The Netherlands (1993)

  • Encarnação, L.M.: Multi-level user support through adaptive hypermedia: a highly application-independent help component. In: Proceedings of the 1997 International Conference on Intelligent User Interfaces, pp. 187–194. Orlando, Florida, United States (1997)

  • Eurostat: http://epp.eurostat.ec.europa.eu (2005)

  • Faulkner L., Wick D.: Cross-user analysis: benefits of skill level comparison in usability testing. Interact. Comput. 17(6), 773–786 (2005)

    Article  Google Scholar 

  • Fischer G.: User modeling in human–computer interaction. User Model. User-adapt. Interact. 11(1-2), 65–86 (2001)

    Article  MATH  Google Scholar 

  • Fitts P.M.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47(6), 381–391 (1954) (Reprinted in J. Exp. Psychol.: General, 121(3):262–269, 1992)

    Article  Google Scholar 

  • Fitts P.M.: Perceptual-motor skill learning. In: Melton, A.W. (eds) Categories of Human Learning, Academic Press, New York and London (1964)

    Google Scholar 

  • Fitts P.M., Posner M.I.: Human Performance. Brooks/Cole Publishing Company, Belmont (1968)

    Google Scholar 

  • Frias-Martinez E., Chen S.Y., Macredie R.D., Liu X.: The role of human factors in stereotyping behavior and perception of digital library users: a robust clustering approach. User Model. User-adapt. Interact. 17(3), 305–337 (2007)

    Article  Google Scholar 

  • Ghazarian, A.: A case study of defect introduction mechanisms. In: Proceedings of the 21st International Conference on Advanced Information Systems (CAiSE’09), pp. 156–170. Amsterdam, The Netherlands (2009)

  • Hacker W.: Action theory and occupational psychology. Review of German empirical research since 1987. Ger. J. Psychol. 18(2), 91–120 (1994)

    Google Scholar 

  • Hackos J.T., Redish J.C.: User and task analysis for interface design. Wiley, New York (1998)

    Google Scholar 

  • Hassel L., Hagen E.: Adaptation of an automotive dialogue system to users’ expertise and evaluation of the system. Computers and the Humanities 40(1), 67–85 (2006)

    Google Scholar 

  • Hilbert, D.M., Robinns, J.E., Redmiles, D.F.: Supporting ongoing user involvement in development via expectation-driven event monitoring. In: Tech. Rep. UCI-ICS-97-19, Department of Information and Computer Science, University of California, Irvine (1997)

  • Hilbert D.M., Redmiles D.F.: Extracting usability information from user interface events. ACM Comput. Surv. 32(4), 384–421 (2000)

    Article  Google Scholar 

  • Horvitz, E., Breese, J., Heckerman, D., Hovel, D., Romelse, K.: The Lumière project: Bayesian user modeling for inferring the goals and needs of software users. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 256–265. Madison, WI (1998)

  • Hurst, A., Hudson, S.E., Mankoff, J.: Dynamic detection of novice versus skilled use without a task model. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 271–280. San Jose, California, USA (2007)

  • Jameson, A.: Generalizing the double-stereotype approach: a psychological perspective. In: Proceedings of the Third International Workshop on User Modeling, pp. 69–83. Wadern, Germany (1992)

  • John B.E., Kieras D.E.: Using GOMS for user interface design and evaluation: which technique?. ACM Trans. Comput. Hum. Interact. 3(4), 287–319 (1996)

    Article  Google Scholar 

  • Jokinen, K., Kanto, K.: User expertise modelling and adaptivity in a speech-based e-mail system. In: Proceedings of the 42nd Annual Meeting on Association For Computational Linguistics, vol. 87. Barcelona, Spain (2004)

  • Kobsa, A.: User modeling: recent work, prospects and hazards. In: SchneiderHufschmidt, M., Kühme, T., Malinowski, U. (eds.) Adaptive User Interfaces: Principles and Practice, pp. 111–128. Amsterdam North-Holland (1993)

  • Kobsa A.: Generic user modeling systems. User Model. User-adapt. Interact. 11(1–2), 49–63 (2001)

    Article  MATH  Google Scholar 

  • Kurosu, M., Urokohara, H., Sato, D., Nishimura, T., Yamada, F.: A new quantitative measure for usability testing: NEM (novice expert ratio method). Poster session presented at the annual conference of the usability professionals’ association on humanizing design, Orlando, Florida (2002)

  • Langley P., Simon H.A.: Applications of machine learning and rule induction. Commun. ACM 38(11), 54–64 (1995)

    Article  Google Scholar 

  • Leung S.C., Fulcher J.: Classification of user expertise level by neural networks. Intl. J. Neural Syst. 8(2), 155–171 (1997)

    Article  Google Scholar 

  • Lin H.C., Shafran I., Yuh D., Hager G.D.: Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comput. Aided Surg. 11(5), 220–230 (2006)

    Article  Google Scholar 

  • Mitchel T.M.: Machine Learning. McGrawHill, New York (1997)

    Google Scholar 

  • Neves D.M., Anderson J.R.: Knowledge compilation: mechanisms for the automatization of cognitive skills. In: Anderson, J.R. (eds) Cognitive Skills and Their Acquisition, pp. 57–84. Wiley, Hillsdale (1981)

    Google Scholar 

  • Newell A., Rosenbloom P.: Mechanisms of skill acquisition and the law of practice. In: Anderson, J.R. (eds) Cognitive Skills and Their Acquisition, pp. 2–55. Erlbaum Associates, Hillsdale (1981)

    Google Scholar 

  • Nielsen J.: Usability Engineering. Academic Press, Boston (1993)

    MATH  Google Scholar 

  • Nisbett R.E., Wilson T.D.: Telling more than we can know: verbal reports on mental processes. Psychol. Rev. 84, 231–259 (1977)

    Article  Google Scholar 

  • Norman D.: Design of Everyday Things. Doubleday, New York (1988)

    Google Scholar 

  • Olson J.R., Olson G.M.: The growth of cognitive modeling in human-computer interaction since GOMS. In: Baecker, R.M., Grudin, J., Buxton, W.A., Greenberg, S. (eds) Human-Computer Interaction: Toward the Year 2000, pp. 603–625. Morgan Kaufmann Publishers, San Francisco (1995)

    Google Scholar 

  • Phillips J.G., Triggs T.J.: Characteristics of cursor trajectories controlled by the computer mouse. Ergonomics 44, 527–536 (2001)

    Article  Google Scholar 

  • Popovic, V.: Novice and Expert User Models, PhD dissertation, University of Sydney (1998)

  • Provost, F.J., Fawcett, T., Kohavi, R.: The Case against accuracy estimation for comparing induction algorithms. In: Proceedings of the Fifteenth International Conference on Machine Learning, pp. 445–453. Madison, Wisconsin, USA (1998)

  • Quinlan J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, Los Altos (1993)

    Google Scholar 

  • Rasmussen J.: Skills, rules and knowledge; signals, signs and symbols, and other distinctions in human performance models. IEEE Trans. Syst. Man Cybern. 13(3), 257–266 (1983)

    Google Scholar 

  • Rich E.: Users are individuals: individualizing user models. Int. J. Hum. Comput. Stud. 51(2), 323–338 (1999)

    Article  MathSciNet  Google Scholar 

  • Richards C., Rosen J., Hannaford B., Pellegrini C., Sinanan M.: Skills evaluation in minimally invasive surgery using force/torque signatures. Surg. Endosc. 14(9), 791–798 (2000)

    Article  Google Scholar 

  • Roberts M.: Integrating the Mind. Routledge, London (2007)

    Google Scholar 

  • Rosen J., Hannaford B., Richards C.G., Sinanan M.N.: Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans. Biomed. Eng. 48(5), 579–591 (2001)

    Article  Google Scholar 

  • Sanderson P.M., Fisher C.: Exploratory sequential data analysis: foundations. Hum. Comput. Interact. Special Issue ESDA 9(3–4), 251–317 (1994)

    Google Scholar 

  • Santos P.J., Badre A.N.: Automatic chunk detection in human-computer interaction. In: Costabile, M.F., Catarci, T., Levialdi, S., Santucci, G. (eds) Proceedings of the Workshop on Advanced Visual Interfaces, pp. 69–77. Bari, Italy (1994)

    Chapter  Google Scholar 

  • Simon H.A., Gilmartin K.: A simulation of memory for chess positions. Cogn. Psychol. 5, 29–46 (1973)

    Article  Google Scholar 

  • Sukaviriya, P.N., Foley, J.D.: Supporting adaptive interfaces in a knowledge-based user interface environment. In: Proceedings of the 1st International Conference on Intelligent User Interfaces, pp. 107–113. Orlando, Florida, United States (1993)

  • Sun R., Giles C.L.: Sequence learning: from recognition and prediction to sequential decision making”. IEEE Intell. Syst. 16(4), 67–70 (2001)

    Article  Google Scholar 

  • Trumbly J.E., Arnett K.P., Martin M.P.: Performance effect of matching computer interface characteristics and user skill level. Int. J. Man-Mach. Stud. 38(4), 713–724 (1993)

    Article  Google Scholar 

  • Uehling, D.L., Wolf, K.: User action graphing effort (UsAGE). In: Conference on Human Factors in Computing Systems, pp. 290–291. Denver, Colorado, United States (1995)

  • Vaubel K.P., Gettys C.F.: Inferring user expertise for adaptive interfaces. Hum. Comput. Interact. 5(1), 95–117 (1990)

    Article  Google Scholar 

  • Webb G.I., Pazzani M.J., Billsus D.: Machine learning for user modeling. User Model. User-adapt. Interact. 11(1–2), 19–29 (2001)

    Article  MATH  Google Scholar 

  • Witten I.H., Frank E.: Data Minning: Practical Machine Learning Tools and Techniques, 2nd Edn. Morgan Kaufmann, San Francisco (2005)

    MATH  Google Scholar 

  • Zukerman I., Albrecht D.W.: Predictive statistical models for user modeling. User Model. User-adapt. Interact. 11(1–2), 5–18 (2001)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Engineering and Information Technology Department, AmirKabir University of Technology, 15875, Tehran, Iran

    Arin Ghazarian & S. Majid Noorhosseini

Authors
  1. Arin Ghazarian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Majid Noorhosseini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arin Ghazarian.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ghazarian, A., Noorhosseini, S.M. Automatic detection of users’ skill levels using high-frequency user interface events. User Model User-Adap Inter 20, 109–146 (2010). https://doi.org/10.1007/s11257-010-9073-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-010-9073-5

Keywords

Search

Navigation