Sensing keyboard input for computer activity recognition with a smartphone

Computer activities such as writing documents and playing games are becoming more and more popular in our daily life. These activities (especially if identified in a non-intrusive manner) can be used to facilitate context-aware services. In this paper, we propose to recognize computer activities through keyboard input sensing with a smart-phone. Specifically, we first utilize the microphone embedded in a smartphone to sense the acoustic signal of keystrokes on a computer keyboard. We then identify keystrokes using fingerprint identification techniques. The determined keystrokes are then corrected by using the proposed adjacent similarity matrix algorithm. Finally, by fusing both semantic and acoustic features, a classification model is constructed to recognize four typical computer activities: chatting, coding, writing documents, and playing games. We evaluated the proposed approach from multiple aspects in realistic environments. Experimental results validated the effectiveness of our approach.