Assisting with frustration in learning via machine learning and computer vision

Frustration in web-based learning is detected and alleviated through a system that incorporates Machine Learning, Computer Vision and Natural Language Processing. The first stage is determining when a student is experiencing a significant level of frustration. The second stage involves finding and presenting similar alternative content as "tips" to the student as a means of alleviating frustration. This system utilizes a mobile application featuring a web-brower that student use to go to any site, though the intention is to assist students in learning scenarios it is equally applicable to other web-based tasks. While the student is browsing, the app monitors the student using the front-facing camera and performs face detection which returns an ROI that is fed into an emotion detection system. A deep-learning CNN is used to perform the emotion detection yielding one of anger, fear, disgust, surprise, neutral, and happy. If a significate negative emotion is detected the system parses the currently viewed web page for content that is used directly in a search or first passed to an NLP stage. The NLP stage gives the saliency of the most prominent entities in the current web page content. The resulting information is used in a web search to form tips for the user. Real test results are given, and the success and challenges faced are presented along with future avenues of work.