SVSV: Online handwritten signature verification based on sound and vibration

Abstract

Handwritten signature is one of the most important behavioral biometrics and plays an important role in the field of identity verification. It is regarded as a legal means to verify personal identity by administrative and financial institutions. Traditional manual signature verification requires large labor costs and the probability of verification error is relatively high. Nowadays, tablets are often used for signature capturing, which motivates us to explore the feasibility of using tablets for signature verification. In this paper, we propose an online handwriting signature verification system based on sound and vibration (SVSV) generated during the signing process. We develop an application to collect signature-related vibration and sound data. We first extract the time domain features of the sound signal and use Fast Fourier Transform to extract the frequency domain features of the sound data. For vibration data, we use Discrete Cosine Transform for dimensionality reduction and feature extraction. Then we fuse the sound and vibration features. Finally, we design an efficient one-class classifier based on the Convolutional Neural Network to perform signature verification. Through extensive experiments with 12 volunteers, the results show that SVSV is a robust and efficient system with an AUC of $0.984$ and an EER of $0.05$.

Introduction

Nowadays, people are paying more and more attention to property and privacy security. In order to protect the property and privacy security of people, it is necessary to verify their identity. At present, the commonly used verification methods include signature, fingerprint, facial recognition and so on. Among them, Handwritten signature is the most widely used verification method in today’s society. It is regarded by administrative and financial institutions as a legal means to verify personal identity. Moreover, it is often used as a means of identity verification for cheques, bequests, contracts and other important documents. One of the main reasons for its widespread use is that the process of collecting handwritten signatures is convenient, non-invasive and familiar to people since they often use signatures in daily life. So the verification of the signature is very important. Although everyone’s signature has its own characteristics, the handwriting of individual signatures is often easier to forge. But the signing process is very difficult to imitate because people learn to sign at a young age, and after years of practice, the signature becomes an unconscious action [1], [2], [3], which makes the process of handwriting signature have the unique characteristics of extremely difficult to forge. Therefore, in order to improve the accuracy of signature verification, many systems based on the signing process are proposed [4], [5], [6].

There are many existing researches on handwritten signature verification. According to the types of signature data, these works can be divided into two categories: offline signature verification and online signature verification. Offline signature verification is mainly based on image data of the signature, which is acquired after the completion of the signing process, while online signature verification is based on the time sequence dynamic information of the signing process. Due to the additional temporal information, online approaches can achieve higher accuracy than offline approaches [8]. In [6], high frequency audio is played to sense the motion of the hand during the signature process and Convolutional Neural Network (CNN) is used for signature verification. However, this work needs to transmit extra ultrasound signals. Although most people cannot hear ultrasound, long-term exposure to ultrasound makes people feel uncomfortable, and it is pollution for pets that can hear ultrasound [34]. [26] use the built-in inertial sensor of the smart bracelet to sense the dynamic data during the signature process to verify the handwritten signature, but this work requires users to wear the device on their dominant hand, which is used to sign. Unfortunately, according to a survey [36], only $34\%$ of people wear a watch on their dominant hand. Moreover, the existing online signature methods are mainly based on Dynamic Time Warping (DTW) [25], [26] and Recurrent Neural Network (RNN) [5], [27], but most of these methods require large amounts of data to be trained for reasonable performance.

Today, electronic signatures are widely used, such as a tablet for electronic signatures in front of every business window in the bank, and users sign electronically with a stylus on the tablet screen. In order to solve the user authentication problem in this scenario, we propose an online signature verification system based on vibration and sound (SVSV). However, there are multiple challenges to realize SVSV. The first is that the system needs to resist imitation attacks and replay attacks. In an imitation attack, a hacker performs the signature verification by imitating a genuine signature. A replay attack occurs when a malicious user acquires data needed to verify a signature in advance, such as a hacker recording sound of a genuine signature and then verifying the signature by playing back the previously recorded sound. In order to resist these two attacks, our system uses sound data and vibration data for signature verification. In our system, the sound data is the data of the rubbing sound of the pen and the screen during the signature process recorded by the built-in microphone of the tablet. Vibration data refers to data of the accelerometer and gyroscope embedded in the tablet computer, which is generated by the vibration of the tablet caused by the user’s hand movement during the signing process. In order to resist imitation attacks, we extract unique user features from the sound data and vibration data of the user’s signatures, and then design an effective one class classifier based on CNN for each user to perform signature verification. Our system can also effectively resist replay attacks. Sound can be stolen by recording, and it can only be used for replay attack. But the accuracy of using recordings for replay attacks is not high, because the sound generated by the friction between the pen and the screen during the signing process is very weak. SVSV uses the embedded microphone of the tablet to record signature sounds, the embedded microphone is close to the signature area. And it can record not only airborne sounds, but also structure-borne sounds, which makes the sound energy received by SVSV significantly larger and the sound contains more features. Malicious users often have to use extra microphones, which are farther away than the embedded microphone of a tablet, to steal sound data. So the stolen sound tends to have a certain loss. If malware can steal the vibration data collected by accelerometer and gyroscope, it is almost impossible for malicious users to imitate the same vibration effect, and it is difficult for malicious users to reconstruct the vibration mode for replay attack. Therefore, our system can effectively resist imitation attacks and replay attacks.

According to the usage scenarios of our system, the tablet is generally placed on the table rather than held in the hand. The movement of the signing process has a small impact on the inertial sensor of the tablet. In order to obtain more user futures, in addition to the commonly used inertial sensor data, we also fuse sound data generated by friction during the signature process for verification. So another challenge is how to extract unique futures related to the user from the subtle sensor data. In order to extract user features from weak signal, we design dual feature extraction method. First, we perform feature extraction on sound data and vibration data, respectively. We use Fast Fourier Transform (FFT) for sound data and extract features such as spectral centroid, spectral spread, etc. For vibration data, we extract its features by Discrete Cosine Transform (DCT). Then we fuse these two features to get a feature matrix. After feature extraction, we can get unique user features.

We consider some scenarios with a large number of users, such as banks, the number of users is often hundreds of millions, so how to design an effective and applicable classifier is the third challenge. Rodriguez et al. [28] and Amancio et al. [29] introduce a variety of well-known classifiers, including SVM, KNN and so on, which can provide guidance for model selection. In the scenario of a small number of users, we can use the common classification algorithm described in [28], [29] to build a multi classification system. For scenarios with hundreds of millions of users, it is impractical to train a multi classification system. Our solution is to train an exclusive model for each user. So we design an efficient one-class classifier based on CNN for each user to perform signature verification. Each user’s model is stored in the database. The function of the model is to determine whether the test signature is the genuine signature of the registered user of the model. We only need to load the specified user’s exclusive model for signature verification. When adding a new user, we only need to train an exclusive model for the new user.

Finally, in order to improve the user experience, the genuine signatures used for training in the our system should be as few as possible. If users need to collect hundreds of signatures to train the model, they may feel it troublesome and refuse to use the system. This can greatly reduce the user experience. Therefore, the number of genuine signatures used for training should be reduced as much as possible under the premise of ensuring the verification performance of the system in an acceptable range. Solving this challenge requires not only deep-level feature extraction, but also an efficient classifier. We design a CNN to train an efficient one-class classifier for each user. We only need 12 genuine signatures for model training for each user and we can get satisfactory results.

We design and develop an APP for signature data collection and implement SVSV based on a tablet. To evaluate the performance of SVSV, we invite 12 volunteers, including 2 females and 10 males, to collect signature data under different experimental conditions. For every volunteer, there are 70 genuine signatures, 60 random forged signatures and 60 skilled forged signatures. Extensive experiments show that SVSV is an efficient, accurate and robust system.

Our contributions are summarized as follows:

• To the best of our knowledge, we are the first to simultaneously study the features of the vibration and the sound generated by the friction during the signing process based on the tablet. We find that each person’s signature has a unique vibration and sound pattern.

• We propose a novel system, SVSV, for online handwritten signature verification based on sound and vibration, which can be perceived by the microphone and inertial sensors in the tablet.

• We perform effective feature extraction on weak signals. We design an efficient one-class classifier based on deep neural networks for signature verification, and our model only needs 12 genuine signatures for training to get ideal results.

• We evaluate SVSV extensively. Experimental results show that SVSV can achieve an AUC of $0.984$ and an EER of $0.05$. Furthermore, the SVSV shows good robustness in different environments.

Roadmap: The remainder of the paper is organized as follows. Section 2 reviews the related work in handwritten signature verification. Section 3 elaborates the design details of SVSV. Section 4 presents extensive experiments and the corresponding results of each experiment. Finally, Section 5 concludes this paper.